Sep 14 02:52:13.163321 ip-10-0-138-23 systemd[1]: Starting Kubernetes Kubelet...
Sep 14 02:52:13.565166 ip-10-0-138-23 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 02:52:13.565166 ip-10-0-138-23 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 14 02:52:13.565166 ip-10-0-138-23 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 02:52:13.565166 ip-10-0-138-23 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 14 02:52:13.565166 ip-10-0-138-23 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 02:52:13.568207 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.568124    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 14 02:52:13.572452 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572429    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:13.572452 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572447    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:13.572452 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572453    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:13.572452 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572457    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572461    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572464    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572467    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572470    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572473    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572475    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572478    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572480    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572483    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572486    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572488    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572491    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572494    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572497    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572502    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572505    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572507    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572510    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572513    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:13.572618 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572516    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572518    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572521    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572523    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572526    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572529    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572531    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572534    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572536    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572539    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572541    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572544    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572546    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572549    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572554    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572558    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572562    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572565    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572568    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:13.573116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572571    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572574    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572577    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572580    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572584    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572587    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572590    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572593    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572597    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572600    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572602    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572605    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572607    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572610    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572613    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572615    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572618    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572620    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572623    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:13.573582 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572626    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572629    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572631    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572634    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572636    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572639    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572642    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572645    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572648    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572651    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572654    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572657    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572660    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572662    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572665    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572667    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572670    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572673    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572675    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572678    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572680    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:13.574050 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572683    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572686    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572689    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572691    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572694    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572697    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572701    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572704    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572706    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572709    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572711    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572714    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572716    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572719    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572721    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572724    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.572726    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573426    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573433    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:13.574597 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573437    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573440    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573443    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573446    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573448    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573451    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573454    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573458    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573462    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573465    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573468    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573471    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573474    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573477    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573479    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573482    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573485    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573488    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573491    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:13.575088 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573493    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573496    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573498    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573501    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573504    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573506    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573509    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573511    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573514    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573517    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573520    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573523    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573525    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573528    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573531    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573534    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573537    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573539    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573542    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573544    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:13.575663 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573547    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573549    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573552    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573555    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573557    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573560    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573563    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573565    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573568    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573571    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573573    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573576    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573578    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573581    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573583    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573586    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573590    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573593    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573596    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:13.576172 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573598    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573601    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573603    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573606    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573609    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573612    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573616    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573619    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573622    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573625    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573629    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573633    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573636    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573638    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573641    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573644    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573646    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573648    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573651    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:13.576638 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573654    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573656    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573659    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573661    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573664    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573667    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573669    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573672    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573675    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573677    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573679    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573682    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573684    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573687    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573689    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573692    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573694    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573697    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573700    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.573703    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:13.577153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574106    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574112    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574115    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574118    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574121    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574124    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574127    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574130    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574133    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574136    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574139    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574142    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574144    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574147    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574149    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574152    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574155    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574158    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574160    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574163    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:13.577648 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574166    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574169    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574172    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574174    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574177    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574179    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574182    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574185    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574187    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574190    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574192    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574199    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574202    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574204    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574207    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574210    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574212    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574215    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574217    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574220    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:13.578142 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574223    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574225    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574228    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574230    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574233    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574235    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574238    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574241    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574245    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574255    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574259    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574262    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574264    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574268    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574272    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574275    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574278    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574281    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574284    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:13.578669 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574286    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574289    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574292    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574294    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574298    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574301    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574304    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574307    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574309    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574312    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574314    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574317    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574319    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574322    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574325    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574327    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574330    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574332    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574334    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574337    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:13.579148 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574339    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574342    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574344    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574348    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574351    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574354    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574356    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574359    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574361    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574364    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574366    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574370    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574373    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574376    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574378    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574380    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574385    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574387    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574390    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.574392    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:13.579630 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574459    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574504    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574512    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574517    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574521    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574525    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574529    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574533    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574536    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574539    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574543    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574546    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574549    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574552    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574555    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574558    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574561    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574564    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574569    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574571    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574574    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574577    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574581    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574584    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 14 02:52:13.580126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574587    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574590    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574593    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574596    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574601    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574604    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574608    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574611    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574616    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574619    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574621    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574624    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574627    2580 flags.go:64] FLAG: --enable-server="true"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574630    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574634    2580 flags.go:64] FLAG: --event-burst="100"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574637    2580 flags.go:64] FLAG: --event-qps="50"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574640    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574643    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574646    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574650    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574653    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574656    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574658    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574661    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574664    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 14 02:52:13.580711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574667    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574670    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574673    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574676    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574679    2580 flags.go:64] FLAG: --feature-gates=""
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574682    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574685    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574688    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574691    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574694    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574697    2580 flags.go:64] FLAG: --help="false"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574701    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574704    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574707    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574710    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574714    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574717    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574720    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574723    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574726    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574729    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574731    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574734    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574737    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 14 02:52:13.581351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574740    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574743    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574746    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574749    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574752    2580 flags.go:64] FLAG: --lock-file=""
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574755    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574757    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574760    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574766    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574769    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574772    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574775    2580 flags.go:64] FLAG: --logging-format="text"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574777    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574781    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574783    2580 flags.go:64] FLAG: --manifest-url=""
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574786    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574790    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574793    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574797    2580 flags.go:64] FLAG: --max-pods="110"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574806    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574809    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574812    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574815    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574818    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574821    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 14 02:52:13.581971 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574824    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574831    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574834    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574837    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574840    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574843    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574848    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574851    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574854    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574856    2580 flags.go:64] FLAG: --port="10250"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574859    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574862    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02cb74db6108d5c3e"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574865    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574868    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574871    2580 flags.go:64] FLAG: --register-node="true"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574874    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574878    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574881    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574884    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574887    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574891    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574894    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574897    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574900    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574902    2580 flags.go:64] FLAG: --runonce="false"
Sep 14 02:52:13.582621 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574905    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574909    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574914    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574917    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574920    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574923    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574926    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574930    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574933    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574936    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574939    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574941    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574945    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574948    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574950    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574956    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574958    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574962    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574966    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574969    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574972    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574975    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574977    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574980    2580 flags.go:64] FLAG: --v="2"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574984    2580 flags.go:64] FLAG: --version="false"
Sep 14 02:52:13.583297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574989    2580 flags.go:64] FLAG: --vmodule=""
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574993    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.574996    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575121    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575125    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575130    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575134    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575136    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575141    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575146    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575149    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575151    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575154    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575157    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575160    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575163    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575166    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575170    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575173    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:13.583899 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575176    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575179    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575182    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575184    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575187    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575189    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575192    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575194    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575197    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575199    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575202    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575204    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575207    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575210    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575213    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575216    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575218    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575221    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575223    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575226    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:13.584396 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575228    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575233    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575236    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575239    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575242    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575244    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575247    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575249    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575252    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575255    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575258    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575260    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575263    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575265    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575268    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575271    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575273    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575277    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575280    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575283    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:13.584877 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575286    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575289    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575292    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575295    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575298    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575302    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575305    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575309    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575312    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575314    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575317    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575319    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575322    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575326    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575330    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575332    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575335    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575337    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575340    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575343    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:13.585372 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575345    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575348    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575350    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575353    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575356    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575358    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575361    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575364    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575366    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575369    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575371    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575374    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575377    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575379    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575382    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575384    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575387    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575389    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575392    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:13.585871 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575394    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575397    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575400    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.575403    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.576311    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.583852    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.583867    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.584737    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.584745    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584809    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584814    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584817    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584820    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584823    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584826    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:13.586349 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584830    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584832    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584835    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584838    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584840    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584843    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584846    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584848    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584851    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584853    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584856    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584859    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584862    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584864    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584867    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584869    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584872    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584875    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584879    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584882    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:13.586723 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584885    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584887    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584890    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584893    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584897    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584899    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584902    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584905    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584907    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584910    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584912    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584915    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584918    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584920    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584923    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584926    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584929    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584931    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584934    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:13.587275 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584938    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584942    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584946    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584949    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584951    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584954    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584956    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584959    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584962    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584965    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584968    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584971    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584974    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584977    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584979    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584983    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584986    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584988    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584991    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:13.587768 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584995    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.584999    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585002    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585005    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585008    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585011    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585013    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585016    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585019    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585022    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585025    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585040    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585043    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585046    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585049    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585053    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585055    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585058    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585061    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585063    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:13.588230 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585066    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585069    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585071    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585074    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585076    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585079    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585082    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585085    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585088    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585091    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585093    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585096    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585099    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585101    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585104    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:13.588707 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.585109    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585220    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585224    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585227    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585230    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585233    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585236    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585239    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585242    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585245    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585248    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585251    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585253    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585257    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585260    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585262    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585266    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585268    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585271    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585274    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585277    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:13.589096 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585280    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585282    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585285    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585289    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585291    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585294    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585297    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585300    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585302    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585305    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585308    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585310    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585313    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585315    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585318    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585321    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585324    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585328    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585331    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:13.589598 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585334    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585337    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585339    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585342    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585344    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585347    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585349    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585352    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585355    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585357    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585360    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585362    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585365    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585367    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585370    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585372    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585375    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585379    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585382    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585385    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:13.590058 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585387    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585390    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585392    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585395    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585398    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585400    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585422    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585425    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585429    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585432    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585434    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585437    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585440    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585442    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585445    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585447    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585450    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585452    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585455    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:13.590542 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585457    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585460    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585462    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585466    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585470    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585472    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585475    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585477    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585480    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585483    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585485    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585489    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585491    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585494    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585498    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585501    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585504    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585506    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585509    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:13.591005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585512    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:13.591482 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:13.585515    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:13.591482 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.585519    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 02:52:13.591482 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.585611    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 14 02:52:13.591482 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.587987    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 14 02:52:13.591482 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.589067    2580 server.go:1019] "Starting client certificate rotation"
Sep 14 02:52:13.591482 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.589168    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 02:52:13.591482 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.589209    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 02:52:13.592270 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.592256    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 02:52:13.593404 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.593375    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 02:52:13.606007 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.605991    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 14 02:52:13.609907 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.609891    2580 log.go:25] "Validated CRI v1 image API"
Sep 14 02:52:13.610723 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.610703    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 14 02:52:13.613168 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.613151    2580 fs.go:135] Filesystem UUIDs: map[0f5c74af-ca28-4473-8878-b810daf4b1a9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7fe8926f-8ef0-49e5-9140-8a9278988043:/dev/nvme0n1p4]
Sep 14 02:52:13.613220 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.613185    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 14 02:52:13.619205 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.619086    2580 manager.go:216] Machine: {Timestamp:2026-09-14 02:52:13.617594938 +0000 UTC m=+0.343552980 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099091 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2358016f3e3714d8f9ca2660f9b39c SystemUUID:ec235801-6f3e-3714-d8f9-ca2660f9b39c BootID:95953ff6-be88-44ed-a354-61f1cc2a2030 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fa:82:aa:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fa:82:aa:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:b3:69:28:c2:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 14 02:52:13.619205 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.619185    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 14 02:52:13.619349 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.619260    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 14 02:52:13.619349 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.619322    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 02:52:13.622847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.622823    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 14 02:52:13.622980 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.622850    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-23.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 14 02:52:13.624361 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.624352    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 14 02:52:13.624393 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.624364    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 14 02:52:13.624393 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.624376    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 02:52:13.625074 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.625063    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 02:52:13.626728 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.626718    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 14 02:52:13.630989 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.630975    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 02:52:13.631096 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.631086    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 14 02:52:13.631130 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.631115    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 14 02:52:13.631166 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.631131    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 14 02:52:13.631166 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.631147    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 14 02:52:13.631166 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.631156    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 14 02:52:13.631166 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.631164    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 14 02:52:13.631554 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.631540    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 02:52:13.631636 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.631575    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 02:52:13.633962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.633940    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 14 02:52:13.637731 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.637680    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 14 02:52:13.637804 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.637757    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 14 02:52:13.639975 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.639961    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 14 02:52:13.639975 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.639978    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 14 02:52:13.640113 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.639984    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 14 02:52:13.640113 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.639989    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 14 02:52:13.640113 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.639999    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 14 02:52:13.640113 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.640004    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 14 02:52:13.640113 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.640018    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 14 02:52:13.640113 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.640022    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 14 02:52:13.640113 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.640043    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 14 02:52:13.640113 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.640051    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 14 02:52:13.640113 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.640062    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 14 02:52:13.640113 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.640067    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 14 02:52:13.640981 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.640970    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 14 02:52:13.640981 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.640979    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 14 02:52:13.641235 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.641213    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 14 02:52:13.641313 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.641282    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 14 02:52:13.644632 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.644620    2580 server.go:1268] "Started kubelet"
Sep 14 02:52:13.644710 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.644671    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 14 02:52:13.644710 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.644686    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 14 02:52:13.644803 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.644724    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 14 02:52:13.645265 ip-10-0-138-23 systemd[1]: Started Kubernetes Kubelet.
Sep 14 02:52:13.647773 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.647760    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 14 02:52:13.648059 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.647983    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 14 02:52:13.651597 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.651579    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 14 02:52:13.651804 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.651788    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-23.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 14 02:52:13.653134 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.652045    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-23.ec2.internal.18d510493a30eb7d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-23.ec2.internal,UID:ip-10-0-138-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-23.ec2.internal,},FirstTimestamp:2026-09-14 02:52:13.644581757 +0000 UTC m=+0.370539799,LastTimestamp:2026-09-14 02:52:13.644581757 +0000 UTC m=+0.370539799,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-23.ec2.internal,}"
Sep 14 02:52:13.658412 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.658393    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 14 02:52:13.658503 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.658429    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 14 02:52:13.659336 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.658861    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found"
Sep 14 02:52:13.659336 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659235    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 14 02:52:13.659336 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659238    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 14 02:52:13.659336 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659257    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 14 02:52:13.659336 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659271    2580 factory.go:55] Registering systemd factory
Sep 14 02:52:13.659336 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659283    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 14 02:52:13.659336 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659313    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 14 02:52:13.659336 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659332    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 14 02:52:13.659747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659510    2580 factory.go:153] Registering CRI-O factory
Sep 14 02:52:13.659747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659525    2580 factory.go:223] Registration of the crio container factory successfully
Sep 14 02:52:13.659747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659572    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 14 02:52:13.659747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659591    2580 factory.go:103] Registering Raw factory
Sep 14 02:52:13.659747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659605    2580 manager.go:1196] Started watching for new ooms in manager
Sep 14 02:52:13.660007 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.659993    2580 manager.go:349] Starting recovery of all containers
Sep 14 02:52:13.660683 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.660633    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 14 02:52:13.660836 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.660814    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 14 02:52:13.670211 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.670081    2580 manager.go:354] Recovery completed
Sep 14 02:52:13.675492 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.675476    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:13.675950 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.675935    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mwxlh"
Sep 14 02:52:13.677127 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.677110    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:13.677228 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.677140    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:13.677228 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.677154    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:13.677701 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.677686    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 14 02:52:13.677701 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.677699    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:13.677849 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.677700    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 14 02:52:13.677849 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.677734    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 02:52:13.678388 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.678359    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:13.678388 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.678387    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:13.678519 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.678400    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:13.679937 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.679920    2580 policy_none.go:49] "None policy: Start"
Sep 14 02:52:13.679937 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.679939    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 14 02:52:13.680045 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.679949    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 14 02:52:13.681871 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.681860    2580 policy_none.go:47] "Start"
Sep 14 02:52:13.685479 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.685446    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mwxlh"
Sep 14 02:52:13.686609 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.686548    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-23.ec2.internal.18d510493c217f39  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-23.ec2.internal,UID:ip-10-0-138-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-23.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-23.ec2.internal,},FirstTimestamp:2026-09-14 02:52:13.677125433 +0000 UTC m=+0.403083483,LastTimestamp:2026-09-14 02:52:13.677125433 +0000 UTC m=+0.403083483,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-23.ec2.internal,}"
Sep 14 02:52:13.718590 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.718573    2580 manager.go:337] "Starting Device Plugin manager"
Sep 14 02:52:13.733290 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.718607    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 14 02:52:13.733290 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.718652    2580 server.go:87] "Starting device plugin registration server"
Sep 14 02:52:13.733290 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.718937    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 14 02:52:13.733290 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.718951    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 14 02:52:13.733290 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.719048    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 14 02:52:13.733290 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.719163    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 14 02:52:13.733290 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.719173    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 14 02:52:13.733290 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.719544    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 14 02:52:13.733290 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.719593    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-23.ec2.internal\" not found"
Sep 14 02:52:13.748897 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.748869    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 14 02:52:13.750000 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.749982    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 14 02:52:13.750099 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.750059    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 14 02:52:13.750099 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.750089    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 14 02:52:13.750184 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.750132    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 14 02:52:13.753082 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.753063    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 02:52:13.819948 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.819878    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:13.822324 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.822309    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:13.822401 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.822336    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:13.822401 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.822347    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:13.822401 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.822368    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.838934 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.838917    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.839016 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.838937    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-23.ec2.internal\": node \"ip-10-0-138-23.ec2.internal\" not found"
Sep 14 02:52:13.850554 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.850536    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal"]
Sep 14 02:52:13.850670 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.850598    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:13.851908 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.851894    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:13.851981 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.851924    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:13.851981 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.851938    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:13.854125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.854111    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:13.854253 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.854240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.854301 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.854271    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:13.854778 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.854762    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:13.854857 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.854785    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:13.854857 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.854795    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:13.855417 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.855404    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:13.855470 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.855424    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:13.855470 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.855436    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:13.857448 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.857435    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.857520 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.857461    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:13.858589 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.858334    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:13.858589 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.858359    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:13.858589 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.858372    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:13.859946 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.859928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cce3e77dd74e99dce2bf49f1ab714fd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"cce3e77dd74e99dce2bf49f1ab714fd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.860045 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.859952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5480c88e36bfd2aebb919a8fa1692c84-config\") pod \"kube-apiserver-proxy-ip-10-0-138-23.ec2.internal\" (UID: \"5480c88e36bfd2aebb919a8fa1692c84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.860045 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.859967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cce3e77dd74e99dce2bf49f1ab714fd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"cce3e77dd74e99dce2bf49f1ab714fd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.881585 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.881566    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-23.ec2.internal\" not found" node="ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.885890 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.885876    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-23.ec2.internal\" not found" node="ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.888706 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.888692    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found"
Sep 14 02:52:13.960321 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.960301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cce3e77dd74e99dce2bf49f1ab714fd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"cce3e77dd74e99dce2bf49f1ab714fd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.960408 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.960280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cce3e77dd74e99dce2bf49f1ab714fd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"cce3e77dd74e99dce2bf49f1ab714fd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.960408 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.960352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cce3e77dd74e99dce2bf49f1ab714fd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"cce3e77dd74e99dce2bf49f1ab714fd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.960408 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.960367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5480c88e36bfd2aebb919a8fa1692c84-config\") pod \"kube-apiserver-proxy-ip-10-0-138-23.ec2.internal\" (UID: \"5480c88e36bfd2aebb919a8fa1692c84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.960408 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.960385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5480c88e36bfd2aebb919a8fa1692c84-config\") pod \"kube-apiserver-proxy-ip-10-0-138-23.ec2.internal\" (UID: \"5480c88e36bfd2aebb919a8fa1692c84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.960529 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:13.960418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cce3e77dd74e99dce2bf49f1ab714fd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"cce3e77dd74e99dce2bf49f1ab714fd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:13.989571 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:13.989555    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found"
Sep 14 02:52:14.090478 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.090428    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found"
Sep 14 02:52:14.184178 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.184148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:14.188902 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.188738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:14.190864 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.190847    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found"
Sep 14 02:52:14.291673 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.291651    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found"
Sep 14 02:52:14.393822 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.393765    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found"
Sep 14 02:52:14.494783 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.494762    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found"
Sep 14 02:52:14.524105 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.524085    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 02:52:14.589824 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.589803    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 14 02:52:14.590213 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.589920    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 14 02:52:14.590213 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.589963    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 14 02:52:14.595562 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.595544    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found"
Sep 14 02:52:14.606885 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.606865    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 02:52:14.632182 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.632161    2580 apiserver.go:52] "Watching apiserver"
Sep 14 02:52:14.645402 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.645341    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 14 02:52:14.646450 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.646429    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-qlq7b","kube-system/global-pull-secret-syncer-ggtgm","kube-system/konnectivity-agent-thtkr","openshift-image-registry/node-ca-852l7","openshift-multus/multus-xt4x7","openshift-network-diagnostics/network-check-target-g4jlk","openshift-network-operator/iptables-alerter-9dx7w","openshift-ovn-kubernetes/ovnkube-node-hq4zd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6","openshift-cluster-node-tuning-operator/tuned-rgpn2","openshift-dns/node-resolver-6ps5c","openshift-multus/multus-additional-cni-plugins-vcrlw"]
Sep 14 02:52:14.649005 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.648985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:14.649090 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.649070    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:14.651076 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.651058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:14.651142 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.651113    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:14.653161 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.653139    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:14.655163 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.655145    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8kxlt\""
Sep 14 02:52:14.655267 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.655149    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 14 02:52:14.655267 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.655151    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 14 02:52:14.655420 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.655400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-852l7"
Sep 14 02:52:14.657582 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.657557    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 14 02:52:14.657689 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.657615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.657863 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.657844    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fbmbl\""
Sep 14 02:52:14.657863 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.657860    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 14 02:52:14.657980 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.657950    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 14 02:52:14.658958 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.658940    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 14 02:52:14.659030 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.658948    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:14.659683 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.659665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 14 02:52:14.659796 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.659776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:14.659870 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.659836    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:14.659870 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.659778    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gchdp\""
Sep 14 02:52:14.659984 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.659892    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 14 02:52:14.659984 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.659893    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 14 02:52:14.659984 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.659846    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 14 02:52:14.662119 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.662087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.664053 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.664031    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 14 02:52:14.664148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.664076    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 14 02:52:14.664393 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.664380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.664795 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.664777    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 14 02:52:14.664889 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.664808    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 14 02:52:14.664889 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.664876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-var-lib-cni-bin\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.664998 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.664921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-system-cni-dir\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.664998 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.664946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-hostroot\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.664998 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.664980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fnq7\" (UniqueName: \"kubernetes.io/projected/0da543da-5bad-4aaf-a25b-748ba3cf0a65-kube-api-access-6fnq7\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:14.665169 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0d4b577-26c7-447d-b6ed-5834730b4051-konnectivity-ca\") pod \"konnectivity-agent-thtkr\" (UID: \"b0d4b577-26c7-447d-b6ed-5834730b4051\") " pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:14.665169 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d2fa0b7-cc69-4546-b798-f5114afde64a-serviceca\") pod \"node-ca-852l7\" (UID: \"6d2fa0b7-cc69-4546-b798-f5114afde64a\") " pod="openshift-image-registry/node-ca-852l7"
Sep 14 02:52:14.665169 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jmmd\" (UniqueName: \"kubernetes.io/projected/6d2fa0b7-cc69-4546-b798-f5114afde64a-kube-api-access-5jmmd\") pod \"node-ca-852l7\" (UID: \"6d2fa0b7-cc69-4546-b798-f5114afde64a\") " pod="openshift-image-registry/node-ca-852l7"
Sep 14 02:52:14.665169 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-socket-dir-parent\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665169 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkqw8\" (UniqueName: \"kubernetes.io/projected/6a8fd5b8-4dab-4952-b507-b72939ac15f8-kube-api-access-hkqw8\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665169 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-cnibin\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-run-k8s-cni-cncf-io\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-run-netns\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-conf-dir\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-run-multus-certs\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kggh\" (UniqueName: \"kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh\") pod \"network-check-target-g4jlk\" (UID: \"6e9096c0-240a-460c-a91d-a8a87df60a9a\") " pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:14.665375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75b4ab6d-419f-495a-b2f5-d62971a327d8-kubelet-config\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:14.665375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-cni-dir\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-daemon-config\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-etc-kubernetes\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-var-lib-cni-multus\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-var-lib-kubelet\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:14.665747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a8fd5b8-4dab-4952-b507-b72939ac15f8-cni-binary-copy\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.665747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:14.665747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75b4ab6d-419f-495a-b2f5-d62971a327d8-dbus\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:14.665747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0d4b577-26c7-447d-b6ed-5834730b4051-agent-certs\") pod \"konnectivity-agent-thtkr\" (UID: \"b0d4b577-26c7-447d-b6ed-5834730b4051\") " pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:14.665747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.665470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d2fa0b7-cc69-4546-b798-f5114afde64a-host\") pod \"node-ca-852l7\" (UID: \"6d2fa0b7-cc69-4546-b798-f5114afde64a\") " pod="openshift-image-registry/node-ca-852l7"
Sep 14 02:52:14.666274 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.666255    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 14 02:52:14.666274 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.666270    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 14 02:52:14.666460 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.666267    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 14 02:52:14.666460 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.666450    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 14 02:52:14.666592 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.666466    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qxr9b\""
Sep 14 02:52:14.666592 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.666555    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 14 02:52:14.666702 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.666623    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n9hw9\""
Sep 14 02:52:14.666819 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.666800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.668810 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.668795    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8sw8k\""
Sep 14 02:52:14.669034 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.669018    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 14 02:52:14.669218 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.669187    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 14 02:52:14.669218 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.669196    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6ps5c"
Sep 14 02:52:14.671129 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.671101    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n8p5b\""
Sep 14 02:52:14.671438 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.671422    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 14 02:52:14.671491 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.671425    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 14 02:52:14.671668 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.671655    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.673395 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.673377    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 14 02:52:14.673979 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.673958    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qrkf7\""
Sep 14 02:52:14.674112 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.673978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9dx7w"
Sep 14 02:52:14.674112 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.674047    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 14 02:52:14.675832 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.675811    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 02:52:14.676078 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.676060    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 14 02:52:14.676622 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.676596    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 14 02:52:14.676713 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.676607    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qll24\""
Sep 14 02:52:14.680478 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.678773    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 02:52:14.680478 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.678882    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal"
Sep 14 02:52:14.680478 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.679134    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal"]
Sep 14 02:52:14.680478 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.679405    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 14 02:52:14.688164 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.688145    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal"]
Sep 14 02:52:14.688288 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.688276    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 02:52:14.689364 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.689319    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 02:47:14 +0000 UTC" deadline="2028-05-24 03:21:11.62079901 +0000 UTC"
Sep 14 02:52:14.689418 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.689364    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14832h28m56.931437528s"
Sep 14 02:52:14.703492 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.703475    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fqpl5"
Sep 14 02:52:14.715338 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.715322    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fqpl5"
Sep 14 02:52:14.759842 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.759822    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 14 02:52:14.765628 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-kubernetes\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.765718 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.765718 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-system-cni-dir\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.765718 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-hostroot\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.765895 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fnq7\" (UniqueName: \"kubernetes.io/projected/0da543da-5bad-4aaf-a25b-748ba3cf0a65-kube-api-access-6fnq7\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:14.765895 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jmmd\" (UniqueName: \"kubernetes.io/projected/6d2fa0b7-cc69-4546-b798-f5114afde64a-kube-api-access-5jmmd\") pod \"node-ca-852l7\" (UID: \"6d2fa0b7-cc69-4546-b798-f5114afde64a\") " pod="openshift-image-registry/node-ca-852l7"
Sep 14 02:52:14.765895 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-socket-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.765895 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-system-cni-dir\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.765895 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765816    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-hostroot\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.765895 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-log-socket\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.765895 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-sysctl-d\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.765895 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-run\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-socket-dir-parent\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkqw8\" (UniqueName: \"kubernetes.io/projected/6a8fd5b8-4dab-4952-b507-b72939ac15f8-kube-api-access-hkqw8\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-slash\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.765984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-socket-dir-parent\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-run-netns\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-node-log\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-run-k8s-cni-cncf-io\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-cni-dir\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-daemon-config\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-run-k8s-cni-cncf-io\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-var-lib-openvswitch\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-cni-dir\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-etc-openvswitch\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj4pb\" (UniqueName: \"kubernetes.io/projected/c3944957-e448-4279-834a-1b091a14afe8-kube-api-access-nj4pb\") pod \"node-resolver-6ps5c\" (UID: \"c3944957-e448-4279-834a-1b091a14afe8\") " pod="openshift-dns/node-resolver-6ps5c"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-var-lib-kubelet\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.766316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6087e55-c525-4aa5-831b-8f9e61a4c4e3-iptables-alerter-script\") pod \"iptables-alerter-9dx7w\" (UID: \"f6087e55-c525-4aa5-831b-8f9e61a4c4e3\") " pod="openshift-network-operator/iptables-alerter-9dx7w"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-var-lib-kubelet\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-cni-bin\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxm2m\" (UniqueName: \"kubernetes.io/projected/5e138d77-4b8b-448f-83b2-16675617de9b-kube-api-access-bxm2m\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.766412    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-lib-modules\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-device-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-systemd-units\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.766513    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret podName:75b4ab6d-419f-495a-b2f5-d62971a327d8 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:15.266471172 +0000 UTC m=+1.992429225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret") pod "global-pull-secret-syncer-ggtgm" (UID: "75b4ab6d-419f-495a-b2f5-d62971a327d8") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-sys\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-kubelet\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-sysctl-conf\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-cnibin\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.767244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0d4b577-26c7-447d-b6ed-5834730b4051-konnectivity-ca\") pod \"konnectivity-agent-thtkr\" (UID: \"b0d4b577-26c7-447d-b6ed-5834730b4051\") " pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d2fa0b7-cc69-4546-b798-f5114afde64a-serviceca\") pod \"node-ca-852l7\" (UID: \"6d2fa0b7-cc69-4546-b798-f5114afde64a\") " pod="openshift-image-registry/node-ca-852l7"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-daemon-config\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5e138d77-4b8b-448f-83b2-16675617de9b-env-overrides\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-modprobe-d\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-tuned\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd66w\" (UniqueName: \"kubernetes.io/projected/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-kube-api-access-qd66w\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3944957-e448-4279-834a-1b091a14afe8-hosts-file\") pod \"node-resolver-6ps5c\" (UID: \"c3944957-e448-4279-834a-1b091a14afe8\") " pod="openshift-dns/node-resolver-6ps5c"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.766987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6087e55-c525-4aa5-831b-8f9e61a4c4e3-host-slash\") pod \"iptables-alerter-9dx7w\" (UID: \"f6087e55-c525-4aa5-831b-8f9e61a4c4e3\") " pod="openshift-network-operator/iptables-alerter-9dx7w"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgw52\" (UniqueName: \"kubernetes.io/projected/f6087e55-c525-4aa5-831b-8f9e61a4c4e3-kube-api-access-qgw52\") pod \"iptables-alerter-9dx7w\" (UID: \"f6087e55-c525-4aa5-831b-8f9e61a4c4e3\") " pod="openshift-network-operator/iptables-alerter-9dx7w"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-cnibin\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-run-netns\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-cnibin\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-conf-dir\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-run-netns\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-run-multus-certs\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kggh\" (UniqueName: \"kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh\") pod \"network-check-target-g4jlk\" (UID: \"6e9096c0-240a-460c-a91d-a8a87df60a9a\") " pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-multus-conf-dir\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.768125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d2fa0b7-cc69-4546-b798-f5114afde64a-serviceca\") pod \"node-ca-852l7\" (UID: \"6d2fa0b7-cc69-4546-b798-f5114afde64a\") " pod="openshift-image-registry/node-ca-852l7"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75b4ab6d-419f-495a-b2f5-d62971a327d8-kubelet-config\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-run-multus-certs\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-etc-selinux\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0d4b577-26c7-447d-b6ed-5834730b4051-konnectivity-ca\") pod \"konnectivity-agent-thtkr\" (UID: \"b0d4b577-26c7-447d-b6ed-5834730b4051\") " pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-etc-kubernetes\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75b4ab6d-419f-495a-b2f5-d62971a327d8-kubelet-config\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-sys-fs\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-etc-kubernetes\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-sysconfig\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-var-lib-kubelet\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3944957-e448-4279-834a-1b091a14afe8-tmp-dir\") pod \"node-resolver-6ps5c\" (UID: \"c3944957-e448-4279-834a-1b091a14afe8\") " pod="openshift-dns/node-resolver-6ps5c"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk9vq\" (UniqueName: \"kubernetes.io/projected/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-kube-api-access-rk9vq\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.768847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-var-lib-cni-multus\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc7ph\" (UniqueName: \"kubernetes.io/projected/9716fa43-32fc-4ab3-a335-d9bf57fbe589-kube-api-access-dc7ph\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e138d77-4b8b-448f-83b2-16675617de9b-ovn-node-metrics-cert\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-host\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a8fd5b8-4dab-4952-b507-b72939ac15f8-cni-binary-copy\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-var-lib-cni-multus\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75b4ab6d-419f-495a-b2f5-d62971a327d8-dbus\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.767724    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-registration-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.767771    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs podName:0da543da-5bad-4aaf-a25b-748ba3cf0a65 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:15.26775686 +0000 UTC m=+1.993714897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs") pod "network-metrics-daemon-qlq7b" (UID: "0da543da-5bad-4aaf-a25b-748ba3cf0a65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5e138d77-4b8b-448f-83b2-16675617de9b-ovnkube-config\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75b4ab6d-419f-495a-b2f5-d62971a327d8-dbus\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-systemd\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-tmp\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0d4b577-26c7-447d-b6ed-5834730b4051-agent-certs\") pod \"konnectivity-agent-thtkr\" (UID: \"b0d4b577-26c7-447d-b6ed-5834730b4051\") " pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d2fa0b7-cc69-4546-b798-f5114afde64a-host\") pod \"node-ca-852l7\" (UID: \"6d2fa0b7-cc69-4546-b798-f5114afde64a\") " pod="openshift-image-registry/node-ca-852l7"
Sep 14 02:52:14.769444 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.767953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d2fa0b7-cc69-4546-b798-f5114afde64a-host\") pod \"node-ca-852l7\" (UID: \"6d2fa0b7-cc69-4546-b798-f5114afde64a\") " pod="openshift-image-registry/node-ca-852l7"
Sep 14 02:52:14.770394 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.768012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-run-ovn\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.770394 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.768062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-cni-netd\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.770394 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.768092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.770394 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.768122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-var-lib-cni-bin\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.770394 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.768155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-run-systemd\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.770394 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.768194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-run-openvswitch\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.770394 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.768202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a8fd5b8-4dab-4952-b507-b72939ac15f8-host-var-lib-cni-bin\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.770394 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.768218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5e138d77-4b8b-448f-83b2-16675617de9b-ovnkube-script-lib\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.770394 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.768242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a8fd5b8-4dab-4952-b507-b72939ac15f8-cni-binary-copy\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.770394 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.768312    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 14 02:52:14.771716 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.771698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0d4b577-26c7-447d-b6ed-5834730b4051-agent-certs\") pod \"konnectivity-agent-thtkr\" (UID: \"b0d4b577-26c7-447d-b6ed-5834730b4051\") " pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:14.775115 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.775080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jmmd\" (UniqueName: \"kubernetes.io/projected/6d2fa0b7-cc69-4546-b798-f5114afde64a-kube-api-access-5jmmd\") pod \"node-ca-852l7\" (UID: \"6d2fa0b7-cc69-4546-b798-f5114afde64a\") " pod="openshift-image-registry/node-ca-852l7"
Sep 14 02:52:14.775800 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.775779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkqw8\" (UniqueName: \"kubernetes.io/projected/6a8fd5b8-4dab-4952-b507-b72939ac15f8-kube-api-access-hkqw8\") pod \"multus-xt4x7\" (UID: \"6a8fd5b8-4dab-4952-b507-b72939ac15f8\") " pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:14.775913 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.775872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fnq7\" (UniqueName: \"kubernetes.io/projected/0da543da-5bad-4aaf-a25b-748ba3cf0a65-kube-api-access-6fnq7\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:14.779613 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.779594    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:14.779710 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.779618    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:14.779710 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.779631    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8kggh for pod openshift-network-diagnostics/network-check-target-g4jlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:14.779710 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:14.779702    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh podName:6e9096c0-240a-460c-a91d-a8a87df60a9a nodeName:}" failed. No retries permitted until 2026-09-14 02:52:15.279676991 +0000 UTC m=+2.005635028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8kggh" (UniqueName: "kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh") pod "network-check-target-g4jlk" (UID: "6e9096c0-240a-460c-a91d-a8a87df60a9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:14.869039 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-etc-selinux\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-sys-fs\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-sysconfig\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-var-lib-kubelet\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3944957-e448-4279-834a-1b091a14afe8-tmp-dir\") pod \"node-resolver-6ps5c\" (UID: \"c3944957-e448-4279-834a-1b091a14afe8\") " pod="openshift-dns/node-resolver-6ps5c"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk9vq\" (UniqueName: \"kubernetes.io/projected/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-kube-api-access-rk9vq\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc7ph\" (UniqueName: \"kubernetes.io/projected/9716fa43-32fc-4ab3-a335-d9bf57fbe589-kube-api-access-dc7ph\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-sys-fs\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-var-lib-kubelet\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e138d77-4b8b-448f-83b2-16675617de9b-ovn-node-metrics-cert\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-host\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.869281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-registration-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-etc-selinux\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5e138d77-4b8b-448f-83b2-16675617de9b-ovnkube-config\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-systemd\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-tmp\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-run-ovn\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-host\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-cni-netd\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-sysconfig\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3944957-e448-4279-834a-1b091a14afe8-tmp-dir\") pod \"node-resolver-6ps5c\" (UID: \"c3944957-e448-4279-834a-1b091a14afe8\") " pod="openshift-dns/node-resolver-6ps5c"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-registration-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-run-systemd\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-cni-netd\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-run-openvswitch\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-run-ovn\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5e138d77-4b8b-448f-83b2-16675617de9b-ovnkube-script-lib\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-run-systemd\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-kubernetes\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-run-openvswitch\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-socket-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-kubernetes\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-log-socket\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-sysctl-d\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-run\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-slash\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-run-netns\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-socket-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-node-log\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869860    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-log-socket\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-var-lib-openvswitch\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.870962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-etc-openvswitch\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-sysctl-d\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj4pb\" (UniqueName: \"kubernetes.io/projected/c3944957-e448-4279-834a-1b091a14afe8-kube-api-access-nj4pb\") pod \"node-resolver-6ps5c\" (UID: \"c3944957-e448-4279-834a-1b091a14afe8\") " pod="openshift-dns/node-resolver-6ps5c"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-node-log\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.869975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-run\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6087e55-c525-4aa5-831b-8f9e61a4c4e3-iptables-alerter-script\") pod \"iptables-alerter-9dx7w\" (UID: \"f6087e55-c525-4aa5-831b-8f9e61a4c4e3\") " pod="openshift-network-operator/iptables-alerter-9dx7w"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-cni-bin\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5e138d77-4b8b-448f-83b2-16675617de9b-ovnkube-config\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxm2m\" (UniqueName: \"kubernetes.io/projected/5e138d77-4b8b-448f-83b2-16675617de9b-kube-api-access-bxm2m\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-lib-modules\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-systemd\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-device-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-systemd-units\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5e138d77-4b8b-448f-83b2-16675617de9b-ovnkube-script-lib\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-sys\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.871860 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-var-lib-openvswitch\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-kubelet\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9716fa43-32fc-4ab3-a335-d9bf57fbe589-device-dir\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-sysctl-conf\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-lib-modules\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-cnibin\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-etc-openvswitch\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-cnibin\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-kubelet\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-cni-bin\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5e138d77-4b8b-448f-83b2-16675617de9b-env-overrides\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-modprobe-d\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-run-netns\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-tuned\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.872565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd66w\" (UniqueName: \"kubernetes.io/projected/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-kube-api-access-qd66w\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-systemd-units\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3944957-e448-4279-834a-1b091a14afe8-hosts-file\") pod \"node-resolver-6ps5c\" (UID: \"c3944957-e448-4279-834a-1b091a14afe8\") " pod="openshift-dns/node-resolver-6ps5c"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6087e55-c525-4aa5-831b-8f9e61a4c4e3-host-slash\") pod \"iptables-alerter-9dx7w\" (UID: \"f6087e55-c525-4aa5-831b-8f9e61a4c4e3\") " pod="openshift-network-operator/iptables-alerter-9dx7w"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6087e55-c525-4aa5-831b-8f9e61a4c4e3-iptables-alerter-script\") pod \"iptables-alerter-9dx7w\" (UID: \"f6087e55-c525-4aa5-831b-8f9e61a4c4e3\") " pod="openshift-network-operator/iptables-alerter-9dx7w"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgw52\" (UniqueName: \"kubernetes.io/projected/f6087e55-c525-4aa5-831b-8f9e61a4c4e3-kube-api-access-qgw52\") pod \"iptables-alerter-9dx7w\" (UID: \"f6087e55-c525-4aa5-831b-8f9e61a4c4e3\") " pod="openshift-network-operator/iptables-alerter-9dx7w"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-sysctl-conf\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e138d77-4b8b-448f-83b2-16675617de9b-host-slash\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-sys\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-modprobe-d\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3944957-e448-4279-834a-1b091a14afe8-hosts-file\") pod \"node-resolver-6ps5c\" (UID: \"c3944957-e448-4279-834a-1b091a14afe8\") " pod="openshift-dns/node-resolver-6ps5c"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.870899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6087e55-c525-4aa5-831b-8f9e61a4c4e3-host-slash\") pod \"iptables-alerter-9dx7w\" (UID: \"f6087e55-c525-4aa5-831b-8f9e61a4c4e3\") " pod="openshift-network-operator/iptables-alerter-9dx7w"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.871007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.871243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5e138d77-4b8b-448f-83b2-16675617de9b-env-overrides\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.871382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.872384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e138d77-4b8b-448f-83b2-16675617de9b-ovn-node-metrics-cert\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.873148 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.872586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-tmp\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.873678 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.873016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-etc-tuned\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.880193 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.880163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgw52\" (UniqueName: \"kubernetes.io/projected/f6087e55-c525-4aa5-831b-8f9e61a4c4e3-kube-api-access-qgw52\") pod \"iptables-alerter-9dx7w\" (UID: \"f6087e55-c525-4aa5-831b-8f9e61a4c4e3\") " pod="openshift-network-operator/iptables-alerter-9dx7w"
Sep 14 02:52:14.880455 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.880428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd66w\" (UniqueName: \"kubernetes.io/projected/f1c24cda-2081-4571-a2d9-e3b7a8c85f25-kube-api-access-qd66w\") pod \"tuned-rgpn2\" (UID: \"f1c24cda-2081-4571-a2d9-e3b7a8c85f25\") " pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:14.880553 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.880483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc7ph\" (UniqueName: \"kubernetes.io/projected/9716fa43-32fc-4ab3-a335-d9bf57fbe589-kube-api-access-dc7ph\") pod \"aws-ebs-csi-driver-node-lk8k6\" (UID: \"9716fa43-32fc-4ab3-a335-d9bf57fbe589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:14.880553 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.880480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk9vq\" (UniqueName: \"kubernetes.io/projected/788a2e1e-f18e-4f17-990d-8cf2916e2bd0-kube-api-access-rk9vq\") pod \"multus-additional-cni-plugins-vcrlw\" (UID: \"788a2e1e-f18e-4f17-990d-8cf2916e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:14.880759 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.880740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj4pb\" (UniqueName: \"kubernetes.io/projected/c3944957-e448-4279-834a-1b091a14afe8-kube-api-access-nj4pb\") pod \"node-resolver-6ps5c\" (UID: \"c3944957-e448-4279-834a-1b091a14afe8\") " pod="openshift-dns/node-resolver-6ps5c"
Sep 14 02:52:14.880818 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.880756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxm2m\" (UniqueName: \"kubernetes.io/projected/5e138d77-4b8b-448f-83b2-16675617de9b-kube-api-access-bxm2m\") pod \"ovnkube-node-hq4zd\" (UID: \"5e138d77-4b8b-448f-83b2-16675617de9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:14.917739 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:14.917703    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce3e77dd74e99dce2bf49f1ab714fd1.slice/crio-4d4a704ad308eaad2d28ab59360044851b91c5730609c95caeb5693be9f4c7aa WatchSource:0}: Error finding container 4d4a704ad308eaad2d28ab59360044851b91c5730609c95caeb5693be9f4c7aa: Status 404 returned error can't find the container with id 4d4a704ad308eaad2d28ab59360044851b91c5730609c95caeb5693be9f4c7aa
Sep 14 02:52:14.917956 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:14.917937    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5480c88e36bfd2aebb919a8fa1692c84.slice/crio-6b58edcabdc548742dc3707222798f9221ed23ff1dbfc6f82bdac9bd6729f18f WatchSource:0}: Error finding container 6b58edcabdc548742dc3707222798f9221ed23ff1dbfc6f82bdac9bd6729f18f: Status 404 returned error can't find the container with id 6b58edcabdc548742dc3707222798f9221ed23ff1dbfc6f82bdac9bd6729f18f
Sep 14 02:52:14.921702 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.921687    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 02:52:14.984435 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.984408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:14.994140 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:14.994117    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d4b577_26c7_447d_b6ed_5834730b4051.slice/crio-b4b65d6d21b7702b1c238198620a7b8d34ded954bc0a0ba7fdd7c2ebaf70373c WatchSource:0}: Error finding container b4b65d6d21b7702b1c238198620a7b8d34ded954bc0a0ba7fdd7c2ebaf70373c: Status 404 returned error can't find the container with id b4b65d6d21b7702b1c238198620a7b8d34ded954bc0a0ba7fdd7c2ebaf70373c
Sep 14 02:52:14.995845 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.995829    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 02:52:14.999589 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:14.999573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-852l7"
Sep 14 02:52:15.008153 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:15.008130    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2fa0b7_cc69_4546_b798_f5114afde64a.slice/crio-886eec3a1154ff013cbb785c0f4e1b5d9ac98f5e1675ebb34739c7a5a6583613 WatchSource:0}: Error finding container 886eec3a1154ff013cbb785c0f4e1b5d9ac98f5e1675ebb34739c7a5a6583613: Status 404 returned error can't find the container with id 886eec3a1154ff013cbb785c0f4e1b5d9ac98f5e1675ebb34739c7a5a6583613
Sep 14 02:52:15.014560 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.014543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xt4x7"
Sep 14 02:52:15.022244 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:15.022222    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8fd5b8_4dab_4952_b507_b72939ac15f8.slice/crio-17e8520812f2fc2aa6884b7dd2be41932f70d370d99270e0e0de7f23c26691fe WatchSource:0}: Error finding container 17e8520812f2fc2aa6884b7dd2be41932f70d370d99270e0e0de7f23c26691fe: Status 404 returned error can't find the container with id 17e8520812f2fc2aa6884b7dd2be41932f70d370d99270e0e0de7f23c26691fe
Sep 14 02:52:15.028499 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.028482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:15.034589 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.034568    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6"
Sep 14 02:52:15.038376 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:15.038276    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e138d77_4b8b_448f_83b2_16675617de9b.slice/crio-fc23b9406910ebfe2d341b24c890b919a88f845e8691e05ac12b0190c641f1c2 WatchSource:0}: Error finding container fc23b9406910ebfe2d341b24c890b919a88f845e8691e05ac12b0190c641f1c2: Status 404 returned error can't find the container with id fc23b9406910ebfe2d341b24c890b919a88f845e8691e05ac12b0190c641f1c2
Sep 14 02:52:15.043460 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:15.043442    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9716fa43_32fc_4ab3_a335_d9bf57fbe589.slice/crio-7c62aa511192165f270285694ac0a381ae1b462289ed77de5b9be5238dd09f60 WatchSource:0}: Error finding container 7c62aa511192165f270285694ac0a381ae1b462289ed77de5b9be5238dd09f60: Status 404 returned error can't find the container with id 7c62aa511192165f270285694ac0a381ae1b462289ed77de5b9be5238dd09f60
Sep 14 02:52:15.067012 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.066986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rgpn2"
Sep 14 02:52:15.077179 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:15.077155    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c24cda_2081_4571_a2d9_e3b7a8c85f25.slice/crio-299d216f95204e366fab6390cbfc98f0678f865f4bc25544473a0469ee10a0ae WatchSource:0}: Error finding container 299d216f95204e366fab6390cbfc98f0678f865f4bc25544473a0469ee10a0ae: Status 404 returned error can't find the container with id 299d216f95204e366fab6390cbfc98f0678f865f4bc25544473a0469ee10a0ae
Sep 14 02:52:15.087493 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.087474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6ps5c"
Sep 14 02:52:15.095759 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:15.095741    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3944957_e448_4279_834a_1b091a14afe8.slice/crio-c73e8abccc75f21ac7bcbcbd4d70c1fe285b3402a24bb3e639d34872ab0814ec WatchSource:0}: Error finding container c73e8abccc75f21ac7bcbcbd4d70c1fe285b3402a24bb3e639d34872ab0814ec: Status 404 returned error can't find the container with id c73e8abccc75f21ac7bcbcbd4d70c1fe285b3402a24bb3e639d34872ab0814ec
Sep 14 02:52:15.104494 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.104477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vcrlw"
Sep 14 02:52:15.110496 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.110482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9dx7w"
Sep 14 02:52:15.112446 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:15.112419    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788a2e1e_f18e_4f17_990d_8cf2916e2bd0.slice/crio-b9efac98dbd77c2bc942099660432ba778e3fdbfa837ea9847ffa732cbc922bd WatchSource:0}: Error finding container b9efac98dbd77c2bc942099660432ba778e3fdbfa837ea9847ffa732cbc922bd: Status 404 returned error can't find the container with id b9efac98dbd77c2bc942099660432ba778e3fdbfa837ea9847ffa732cbc922bd
Sep 14 02:52:15.119459 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:15.119441    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6087e55_c525_4aa5_831b_8f9e61a4c4e3.slice/crio-f1955283dd88c00b53179c308cc00fb85e272b047a6e1f4bf9e434fed93f8dca WatchSource:0}: Error finding container f1955283dd88c00b53179c308cc00fb85e272b047a6e1f4bf9e434fed93f8dca: Status 404 returned error can't find the container with id f1955283dd88c00b53179c308cc00fb85e272b047a6e1f4bf9e434fed93f8dca
Sep 14 02:52:15.121078 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:15.121059    2580 kuberuntime_manager.go:1459] "Unhandled Error" err="container iptables-alerter start failed in pod iptables-alerter-9dx7w_openshift-network-operator(f6087e55-c525-4aa5-831b-8f9e61a4c4e3): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 14 02:52:15.121138 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:15.121100    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-9dx7w" podUID="f6087e55-c525-4aa5-831b-8f9e61a4c4e3"
Sep 14 02:52:15.274202 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.274112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:15.274202 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.274184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:15.274381 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:15.274292    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:15.274381 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:15.274327    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:15.274381 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:15.274371    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs podName:0da543da-5bad-4aaf-a25b-748ba3cf0a65 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:16.274353851 +0000 UTC m=+3.000311905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs") pod "network-metrics-daemon-qlq7b" (UID: "0da543da-5bad-4aaf-a25b-748ba3cf0a65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:15.274486 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:15.274388    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret podName:75b4ab6d-419f-495a-b2f5-d62971a327d8 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:16.274380279 +0000 UTC m=+3.000338311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret") pod "global-pull-secret-syncer-ggtgm" (UID: "75b4ab6d-419f-495a-b2f5-d62971a327d8") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:15.374667 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.374626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kggh\" (UniqueName: \"kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh\") pod \"network-check-target-g4jlk\" (UID: \"6e9096c0-240a-460c-a91d-a8a87df60a9a\") " pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:15.374858 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:15.374798    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:15.374858 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:15.374819    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:15.374858 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:15.374830    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8kggh for pod openshift-network-diagnostics/network-check-target-g4jlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:15.374974 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:15.374884    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh podName:6e9096c0-240a-460c-a91d-a8a87df60a9a nodeName:}" failed. No retries permitted until 2026-09-14 02:52:16.37486834 +0000 UTC m=+3.100826379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kggh" (UniqueName: "kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh") pod "network-check-target-g4jlk" (UID: "6e9096c0-240a-460c-a91d-a8a87df60a9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:15.717933 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.717847    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 02:47:15 +0000 UTC" deadline="2028-02-14 13:35:12.701746326 +0000 UTC"
Sep 14 02:52:15.717933 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.717888    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12442h42m56.983863681s"
Sep 14 02:52:15.785805 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.785500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" event={"ID":"5480c88e36bfd2aebb919a8fa1692c84","Type":"ContainerStarted","Data":"6b58edcabdc548742dc3707222798f9221ed23ff1dbfc6f82bdac9bd6729f18f"}
Sep 14 02:52:15.811862 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.811823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6ps5c" event={"ID":"c3944957-e448-4279-834a-1b091a14afe8","Type":"ContainerStarted","Data":"c73e8abccc75f21ac7bcbcbd4d70c1fe285b3402a24bb3e639d34872ab0814ec"}
Sep 14 02:52:15.815369 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.815339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rgpn2" event={"ID":"f1c24cda-2081-4571-a2d9-e3b7a8c85f25","Type":"ContainerStarted","Data":"299d216f95204e366fab6390cbfc98f0678f865f4bc25544473a0469ee10a0ae"}
Sep 14 02:52:15.819169 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.819131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6" event={"ID":"9716fa43-32fc-4ab3-a335-d9bf57fbe589","Type":"ContainerStarted","Data":"7c62aa511192165f270285694ac0a381ae1b462289ed77de5b9be5238dd09f60"}
Sep 14 02:52:15.826820 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.826791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xt4x7" event={"ID":"6a8fd5b8-4dab-4952-b507-b72939ac15f8","Type":"ContainerStarted","Data":"17e8520812f2fc2aa6884b7dd2be41932f70d370d99270e0e0de7f23c26691fe"}
Sep 14 02:52:15.844917 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.844600    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" event={"ID":"cce3e77dd74e99dce2bf49f1ab714fd1","Type":"ContainerStarted","Data":"4d4a704ad308eaad2d28ab59360044851b91c5730609c95caeb5693be9f4c7aa"}
Sep 14 02:52:15.857820 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.857789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9dx7w" event={"ID":"f6087e55-c525-4aa5-831b-8f9e61a4c4e3","Type":"ContainerStarted","Data":"f1955283dd88c00b53179c308cc00fb85e272b047a6e1f4bf9e434fed93f8dca"}
Sep 14 02:52:15.861712 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:15.861634    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0444a9580f430c7c6b52f78667a80a82198c825805f1f2004fdfdf9aed2b95f5\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-9dx7w" podUID="f6087e55-c525-4aa5-831b-8f9e61a4c4e3"
Sep 14 02:52:15.864153 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.864132    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 02:52:15.872013 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.871988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcrlw" event={"ID":"788a2e1e-f18e-4f17-990d-8cf2916e2bd0","Type":"ContainerStarted","Data":"b9efac98dbd77c2bc942099660432ba778e3fdbfa837ea9847ffa732cbc922bd"}
Sep 14 02:52:15.888718 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.888691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd" event={"ID":"5e138d77-4b8b-448f-83b2-16675617de9b","Type":"ContainerStarted","Data":"fc23b9406910ebfe2d341b24c890b919a88f845e8691e05ac12b0190c641f1c2"}
Sep 14 02:52:15.899619 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.899592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-852l7" event={"ID":"6d2fa0b7-cc69-4546-b798-f5114afde64a","Type":"ContainerStarted","Data":"886eec3a1154ff013cbb785c0f4e1b5d9ac98f5e1675ebb34739c7a5a6583613"}
Sep 14 02:52:15.905987 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:15.905951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-thtkr" event={"ID":"b0d4b577-26c7-447d-b6ed-5834730b4051","Type":"ContainerStarted","Data":"b4b65d6d21b7702b1c238198620a7b8d34ded954bc0a0ba7fdd7c2ebaf70373c"}
Sep 14 02:52:16.282435 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:16.282393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:16.282631 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:16.282483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:16.282631 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:16.282614    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:16.282757 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:16.282676    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs podName:0da543da-5bad-4aaf-a25b-748ba3cf0a65 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:18.282655949 +0000 UTC m=+5.008613984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs") pod "network-metrics-daemon-qlq7b" (UID: "0da543da-5bad-4aaf-a25b-748ba3cf0a65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:16.283481 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:16.283095    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:16.283481 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:16.283165    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret podName:75b4ab6d-419f-495a-b2f5-d62971a327d8 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:18.283148131 +0000 UTC m=+5.009106164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret") pod "global-pull-secret-syncer-ggtgm" (UID: "75b4ab6d-419f-495a-b2f5-d62971a327d8") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:16.382914 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:16.382874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kggh\" (UniqueName: \"kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh\") pod \"network-check-target-g4jlk\" (UID: \"6e9096c0-240a-460c-a91d-a8a87df60a9a\") " pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:16.383139 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:16.383120    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:16.383212 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:16.383181    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:16.383212 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:16.383198    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8kggh for pod openshift-network-diagnostics/network-check-target-g4jlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:16.383322 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:16.383252    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh podName:6e9096c0-240a-460c-a91d-a8a87df60a9a nodeName:}" failed. No retries permitted until 2026-09-14 02:52:18.383233448 +0000 UTC m=+5.109191488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kggh" (UniqueName: "kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh") pod "network-check-target-g4jlk" (UID: "6e9096c0-240a-460c-a91d-a8a87df60a9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:16.718350 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:16.718234    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 02:47:15 +0000 UTC" deadline="2028-03-26 03:10:33.192599293 +0000 UTC"
Sep 14 02:52:16.718350 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:16.718300    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13416h18m16.47430422s"
Sep 14 02:52:16.751095 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:16.751067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:16.751250 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:16.751224    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:16.751733 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:16.751697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:16.751831 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:16.751816    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:16.751903 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:16.751889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:16.752015 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:16.751977    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:17.489404 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:17.489367    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 02:52:18.299654 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:18.299617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:18.300156 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:18.299698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:18.300156 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:18.299860    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:18.300156 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:18.299919    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret podName:75b4ab6d-419f-495a-b2f5-d62971a327d8 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:22.299900438 +0000 UTC m=+9.025858485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret") pod "global-pull-secret-syncer-ggtgm" (UID: "75b4ab6d-419f-495a-b2f5-d62971a327d8") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:18.300358 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:18.300339    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:18.300414 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:18.300399    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs podName:0da543da-5bad-4aaf-a25b-748ba3cf0a65 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:22.3003808 +0000 UTC m=+9.026338848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs") pod "network-metrics-daemon-qlq7b" (UID: "0da543da-5bad-4aaf-a25b-748ba3cf0a65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:18.400932 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:18.400893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kggh\" (UniqueName: \"kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh\") pod \"network-check-target-g4jlk\" (UID: \"6e9096c0-240a-460c-a91d-a8a87df60a9a\") " pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:18.401146 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:18.401128    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:18.401230 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:18.401154    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:18.401230 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:18.401168    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8kggh for pod openshift-network-diagnostics/network-check-target-g4jlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:18.401230 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:18.401224    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh podName:6e9096c0-240a-460c-a91d-a8a87df60a9a nodeName:}" failed. No retries permitted until 2026-09-14 02:52:22.401204948 +0000 UTC m=+9.127163021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kggh" (UniqueName: "kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh") pod "network-check-target-g4jlk" (UID: "6e9096c0-240a-460c-a91d-a8a87df60a9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:18.751179 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:18.751145    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:18.751367 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:18.751309    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:18.751784 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:18.751756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:18.751888 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:18.751859    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:18.752012 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:18.751913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:18.752012 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:18.751988    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:20.750311 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:20.750269    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:20.750772 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:20.750400    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:20.750847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:20.750791    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:20.750927 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:20.750897    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:20.750987 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:20.750947    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:20.751124 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:20.751024    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:22.330693 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:22.330648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:22.331345 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:22.330840    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:22.331345 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:22.330839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:22.331345 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:22.330913    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret podName:75b4ab6d-419f-495a-b2f5-d62971a327d8 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:30.330896888 +0000 UTC m=+17.056854926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret") pod "global-pull-secret-syncer-ggtgm" (UID: "75b4ab6d-419f-495a-b2f5-d62971a327d8") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:22.331345 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:22.330978    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:22.331345 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:22.331038    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs podName:0da543da-5bad-4aaf-a25b-748ba3cf0a65 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:30.331020285 +0000 UTC m=+17.056978332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs") pod "network-metrics-daemon-qlq7b" (UID: "0da543da-5bad-4aaf-a25b-748ba3cf0a65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:22.431280 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:22.431241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kggh\" (UniqueName: \"kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh\") pod \"network-check-target-g4jlk\" (UID: \"6e9096c0-240a-460c-a91d-a8a87df60a9a\") " pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:22.431469 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:22.431442    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:22.431543 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:22.431477    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:22.431543 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:22.431491    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8kggh for pod openshift-network-diagnostics/network-check-target-g4jlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:22.431658 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:22.431563    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh podName:6e9096c0-240a-460c-a91d-a8a87df60a9a nodeName:}" failed. No retries permitted until 2026-09-14 02:52:30.431537046 +0000 UTC m=+17.157495085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kggh" (UniqueName: "kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh") pod "network-check-target-g4jlk" (UID: "6e9096c0-240a-460c-a91d-a8a87df60a9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:22.751777 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:22.751284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:22.751777 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:22.751327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:22.751777 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:22.751284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:22.751777 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:22.751419    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:22.751777 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:22.751509    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:22.751777 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:22.751586    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:24.751220 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:24.751107    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:24.751220 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:24.751140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:24.751220 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:24.751140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:24.751763 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:24.751224    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:24.751763 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:24.751329    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:24.751763 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:24.751406    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:26.750504 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:26.750469    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:26.750946 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:26.750487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:26.750946 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:26.750586    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:26.750946 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:26.750577    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:26.750946 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:26.750671    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:26.750946 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:26.750752    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:28.750760 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:28.750721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:28.751238 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:28.750721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:28.751238 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:28.750860    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:28.751238 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:28.750731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:28.751238 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:28.750926    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:28.751238 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:28.751053    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:30.390015 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:30.389981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:30.390458 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:30.390051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:30.390458 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:30.390138    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:30.390458 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:30.390165    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:30.390458 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:30.390202    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs podName:0da543da-5bad-4aaf-a25b-748ba3cf0a65 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:46.390184149 +0000 UTC m=+33.116142187 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs") pod "network-metrics-daemon-qlq7b" (UID: "0da543da-5bad-4aaf-a25b-748ba3cf0a65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:30.390458 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:30.390217    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret podName:75b4ab6d-419f-495a-b2f5-d62971a327d8 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:46.390210622 +0000 UTC m=+33.116168652 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret") pod "global-pull-secret-syncer-ggtgm" (UID: "75b4ab6d-419f-495a-b2f5-d62971a327d8") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:30.491043 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:30.490998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kggh\" (UniqueName: \"kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh\") pod \"network-check-target-g4jlk\" (UID: \"6e9096c0-240a-460c-a91d-a8a87df60a9a\") " pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:30.491194 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:30.491161    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:30.491194 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:30.491188    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:30.491284 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:30.491202    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8kggh for pod openshift-network-diagnostics/network-check-target-g4jlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:30.491284 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:30.491263    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh podName:6e9096c0-240a-460c-a91d-a8a87df60a9a nodeName:}" failed. No retries permitted until 2026-09-14 02:52:46.491245265 +0000 UTC m=+33.217203300 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kggh" (UniqueName: "kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh") pod "network-check-target-g4jlk" (UID: "6e9096c0-240a-460c-a91d-a8a87df60a9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:30.750603 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:30.750526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:30.750780 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:30.750634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:30.750780 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:30.750636    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:30.750780 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:30.750752    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:30.750780 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:30.750770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:30.750985 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:30.750858    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:32.750992 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:32.750959    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:32.751440 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:32.751088    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:32.751440 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:32.751093    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:32.751440 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:32.751175    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:32.751440 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:32.751205    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:32.751440 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:32.751277    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:34.750765 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.750601    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:34.751285 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.750611    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:34.751285 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:34.750867    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:34.751285 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:34.750916    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:34.751285 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.750619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:34.751285 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:34.751020    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:34.958074 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.958018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6ps5c" event={"ID":"c3944957-e448-4279-834a-1b091a14afe8","Type":"ContainerStarted","Data":"b05e992bdbfd59966cc685a8a13ebf25ad8e37979d2c697ad242d46f4e92deb6"}
Sep 14 02:52:34.959874 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.959769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rgpn2" event={"ID":"f1c24cda-2081-4571-a2d9-e3b7a8c85f25","Type":"ContainerStarted","Data":"20a2d16a683313be44f9f3ad383e276fb25c5d2af388a5bb06a11502edce09f4"}
Sep 14 02:52:34.961468 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.961441    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6" event={"ID":"9716fa43-32fc-4ab3-a335-d9bf57fbe589","Type":"ContainerStarted","Data":"5f53de69b77adb5ad02deefe6f9c48f78f82f81e54a2820aa401dcc27dc6032b"}
Sep 14 02:52:34.963219 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.963187    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xt4x7" event={"ID":"6a8fd5b8-4dab-4952-b507-b72939ac15f8","Type":"ContainerStarted","Data":"f5b17e230ff3420a5a66cfcf30c003e7a4713b975b7eefd04c503135b03a66c9"}
Sep 14 02:52:34.964896 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.964874    2580 generic.go:356] "Generic (PLEG): container finished" podID="cce3e77dd74e99dce2bf49f1ab714fd1" containerID="b00ca53c1a848c20b372043ff9d0693a306ff8418dec7db9608aa64f89849ce5" exitCode=0
Sep 14 02:52:34.965001 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.964940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" event={"ID":"cce3e77dd74e99dce2bf49f1ab714fd1","Type":"ContainerDied","Data":"b00ca53c1a848c20b372043ff9d0693a306ff8418dec7db9608aa64f89849ce5"}
Sep 14 02:52:34.967138 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.967113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcrlw" event={"ID":"788a2e1e-f18e-4f17-990d-8cf2916e2bd0","Type":"ContainerStarted","Data":"455a89db8f61bd63e106d5912b376101a35405ece72b5c5a95519e8321efd82b"}
Sep 14 02:52:34.974481 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.974155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd" event={"ID":"5e138d77-4b8b-448f-83b2-16675617de9b","Type":"ContainerStarted","Data":"c855065c2c51738989ee4ffc21f8afcbf85a731cb526c4902cc7066eb3a12951"}
Sep 14 02:52:34.974481 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.974309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd" event={"ID":"5e138d77-4b8b-448f-83b2-16675617de9b","Type":"ContainerStarted","Data":"5936433f93dd8ac9ee585ddbb7fb3e6d2c1723d505204f447b8c33bb59fee738"}
Sep 14 02:52:34.974481 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.974407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd" event={"ID":"5e138d77-4b8b-448f-83b2-16675617de9b","Type":"ContainerStarted","Data":"0e72db214ebaea4efe3e0b2bb10c57e0d3ae09a63109f690839b8145a5c243f0"}
Sep 14 02:52:34.976283 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.976259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-852l7" event={"ID":"6d2fa0b7-cc69-4546-b798-f5114afde64a","Type":"ContainerStarted","Data":"f725396a639001e3ebd539e578442c10bda75963ff633433996fa36346758e2f"}
Sep 14 02:52:34.977874 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.977852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-thtkr" event={"ID":"b0d4b577-26c7-447d-b6ed-5834730b4051","Type":"ContainerStarted","Data":"27e907aac826a990a1fc9195d96c2c174433287788b5c75190a17c16124d28a4"}
Sep 14 02:52:34.979604 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.979556    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6ps5c" podStartSLOduration=1.8640600059999999 podStartE2EDuration="20.979544438s" totalImagesPullingTime="19.115484432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:34.978880077 +0000 UTC m=+21.704838129" watchObservedRunningTime="2026-09-14 02:52:34.979544438 +0000 UTC m=+21.705502489"
Sep 14 02:52:34.979728 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:34.979696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" event={"ID":"5480c88e36bfd2aebb919a8fa1692c84","Type":"ContainerStarted","Data":"c15bec43f53f01535605bb3a43329259a23ed960b2c7c3202c7f16b8bddd6288"}
Sep 14 02:52:35.041496 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:35.041453    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-852l7" podStartSLOduration=2.081265599 podStartE2EDuration="21.041438835s" totalImagesPullingTime="18.960173236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:35.021491375 +0000 UTC m=+21.747449429" watchObservedRunningTime="2026-09-14 02:52:35.041438835 +0000 UTC m=+21.767396955"
Sep 14 02:52:35.041616 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:35.041541    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xt4x7" podStartSLOduration=1.780630676 podStartE2EDuration="21.041531135s" totalImagesPullingTime="19.260900459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:35.041333021 +0000 UTC m=+21.767291074" watchObservedRunningTime="2026-09-14 02:52:35.041531135 +0000 UTC m=+21.767489187"
Sep 14 02:52:35.061297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:35.061263    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rgpn2" podStartSLOduration=1.8855076419999999 podStartE2EDuration="21.061248714s" totalImagesPullingTime="19.175741072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:35.06068568 +0000 UTC m=+21.786643732" watchObservedRunningTime="2026-09-14 02:52:35.061248714 +0000 UTC m=+21.787206766"
Sep 14 02:52:35.117641 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:35.117603    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-thtkr" podStartSLOduration=1.892706516 podStartE2EDuration="21.117589007s" totalImagesPullingTime="19.224882491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:35.117367604 +0000 UTC m=+21.843325656" watchObservedRunningTime="2026-09-14 02:52:35.117589007 +0000 UTC m=+21.843547063"
Sep 14 02:52:35.985255 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:35.984586    2580 generic.go:356] "Generic (PLEG): container finished" podID="788a2e1e-f18e-4f17-990d-8cf2916e2bd0" containerID="455a89db8f61bd63e106d5912b376101a35405ece72b5c5a95519e8321efd82b" exitCode=0
Sep 14 02:52:35.985255 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:35.984656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcrlw" event={"ID":"788a2e1e-f18e-4f17-990d-8cf2916e2bd0","Type":"ContainerDied","Data":"455a89db8f61bd63e106d5912b376101a35405ece72b5c5a95519e8321efd82b"}
Sep 14 02:52:35.990156 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:35.990130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd" event={"ID":"5e138d77-4b8b-448f-83b2-16675617de9b","Type":"ContainerStarted","Data":"fd41824d607e322c99925ec33abb34598736f7cdbafe07af307bb2013b725017"}
Sep 14 02:52:35.990243 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:35.990211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd" event={"ID":"5e138d77-4b8b-448f-83b2-16675617de9b","Type":"ContainerStarted","Data":"6460fefd6a41c4c51cc52e3f1ae7762b1c485fb31b186d9ebf05761881d47fef"}
Sep 14 02:52:35.990293 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:35.990252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd" event={"ID":"5e138d77-4b8b-448f-83b2-16675617de9b","Type":"ContainerStarted","Data":"459391a5e57b3addc1bfbd215ebbc109be5df2f39190c6c4840aac526b97952b"}
Sep 14 02:52:35.992669 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:35.992641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" event={"ID":"cce3e77dd74e99dce2bf49f1ab714fd1","Type":"ContainerStarted","Data":"98e06eb0829ee837fe0db3461aca626083a59b1a89f47519390afdb1ae1e7015"}
Sep 14 02:52:35.997438 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:35.995804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9dx7w" event={"ID":"f6087e55-c525-4aa5-831b-8f9e61a4c4e3","Type":"ContainerStarted","Data":"ff8183248c80bb804f981c12d3fd3fb44b20872067c55e19eb436851c25ce1a7"}
Sep 14 02:52:36.017265 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:36.017227    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" podStartSLOduration=21.0172159 podStartE2EDuration="21.0172159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:35.145054295 +0000 UTC m=+21.871012343" watchObservedRunningTime="2026-09-14 02:52:36.0172159 +0000 UTC m=+22.743174018"
Sep 14 02:52:36.036529 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:36.036512    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 14 02:52:36.053896 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:36.053859    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9dx7w" podStartSLOduration=21.053846988 podStartE2EDuration="21.053846988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-14 02:52:36.035581478 +0000 UTC m=+22.761539529" watchObservedRunningTime="2026-09-14 02:52:36.053846988 +0000 UTC m=+22.779805041"
Sep 14 02:52:36.054057 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:36.054020    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" podStartSLOduration=21.054015978 podStartE2EDuration="21.054015978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:36.053907465 +0000 UTC m=+22.779865518" watchObservedRunningTime="2026-09-14 02:52:36.054015978 +0000 UTC m=+22.779974030"
Sep 14 02:52:36.731299 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:36.731179    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T02:52:36.036524969Z","UUID":"92deddc2-8150-4a01-ad42-e8d742ed9dba","Handler":null,"Name":"","Endpoint":""}
Sep 14 02:52:36.732967 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:36.732939    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 14 02:52:36.733126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:36.732975    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 14 02:52:36.751012 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:36.750986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:36.751012 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:36.751007    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:36.751205 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:36.751008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:36.751205 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:36.751129    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:36.751311 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:36.751253    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:36.751404 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:36.751376    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:37.000400 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:37.000191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6" event={"ID":"9716fa43-32fc-4ab3-a335-d9bf57fbe589","Type":"ContainerStarted","Data":"1ad0b57082d07d644917c17ab688ce792dee90d53c7b84bb9e30b59707b76d56"}
Sep 14 02:52:37.000400 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:37.000378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6" event={"ID":"9716fa43-32fc-4ab3-a335-d9bf57fbe589","Type":"ContainerStarted","Data":"f0c5bea412a7bd45edcb64bb00121fa697e259eb17426d6f519ea844e32fcd58"}
Sep 14 02:52:37.914840 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:37.914812    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:38.006440 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:38.006396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd" event={"ID":"5e138d77-4b8b-448f-83b2-16675617de9b","Type":"ContainerStarted","Data":"25792f9ea0430f811b9f5331f380d1bdf35ecd5169310d5e077397d06836d06e"}
Sep 14 02:52:38.134244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:38.134213    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:38.134848 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:38.134830    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:38.153375 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:38.151614    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk8k6" podStartSLOduration=3.389812418 podStartE2EDuration="24.151596s" totalImagesPullingTime="20.761783582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:37.027858543 +0000 UTC m=+23.753816593" watchObservedRunningTime="2026-09-14 02:52:38.151596 +0000 UTC m=+24.877554052"
Sep 14 02:52:38.750990 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:38.750963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:38.751179 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:38.751074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:38.751179 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:38.751078    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:38.751179 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:38.751081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:38.751179 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:38.751170    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:38.751371 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:38.751261    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:39.009339 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:39.009260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-thtkr"
Sep 14 02:52:40.751317 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:40.751280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:40.751741 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:40.751396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:40.751741 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:40.751408    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:40.751741 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:40.751442    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:40.751741 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:40.751529    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:40.751741 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:40.751620    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:41.018344 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:41.018268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd" event={"ID":"5e138d77-4b8b-448f-83b2-16675617de9b","Type":"ContainerStarted","Data":"48c3455be23ae47d4dd2cf7e4e6beb83675ebc465a677569e772fefbc7500523"}
Sep 14 02:52:42.021852 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.021685    2580 generic.go:356] "Generic (PLEG): container finished" podID="788a2e1e-f18e-4f17-990d-8cf2916e2bd0" containerID="91a38133170943529b8c99cd54c5f84966b653a9ea7afc9600bee49a45d19c4d" exitCode=0
Sep 14 02:52:42.022298 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.021768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcrlw" event={"ID":"788a2e1e-f18e-4f17-990d-8cf2916e2bd0","Type":"ContainerDied","Data":"91a38133170943529b8c99cd54c5f84966b653a9ea7afc9600bee49a45d19c4d"}
Sep 14 02:52:42.022298 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.022109    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:42.022298 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.022175    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:42.022298 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.022222    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:42.037100 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.037079    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:42.037195 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.037181    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:52:42.074659 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.074613    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd" podStartSLOduration=8.559970786000001 podStartE2EDuration="28.074600232s" totalImagesPullingTime="19.514629446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:42.074557356 +0000 UTC m=+28.800515408" watchObservedRunningTime="2026-09-14 02:52:42.074600232 +0000 UTC m=+28.800558284"
Sep 14 02:52:42.750662 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.750629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:42.750793 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.750672    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:42.750793 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.750674    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:42.750793 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:42.750745    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:42.750907 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:42.750852    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:42.750948 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:42.750913    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:42.917227 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.917200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qlq7b"]
Sep 14 02:52:42.922108 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.922085    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ggtgm"]
Sep 14 02:52:42.926684 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:42.926665    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g4jlk"]
Sep 14 02:52:43.024506 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:43.024433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:43.024506 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:43.024452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:43.024506 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:43.024454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:43.025066 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:43.024539    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:43.025066 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:43.024599    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:43.025141 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:43.025122    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:44.027880 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:44.027797    2580 generic.go:356] "Generic (PLEG): container finished" podID="788a2e1e-f18e-4f17-990d-8cf2916e2bd0" containerID="ddb3b8b72f76f15446ffd9113f4de36dd04a9f1761c105b4c5d2433cc6432524" exitCode=0
Sep 14 02:52:44.028367 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:44.027876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcrlw" event={"ID":"788a2e1e-f18e-4f17-990d-8cf2916e2bd0","Type":"ContainerDied","Data":"ddb3b8b72f76f15446ffd9113f4de36dd04a9f1761c105b4c5d2433cc6432524"}
Sep 14 02:52:44.751119 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:44.751026    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:44.751275 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:44.751150    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:44.751275 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:44.751184    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:44.751381 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:44.751289    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:44.751437 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:44.751387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:44.751494 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:44.751476    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:46.034414 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:46.034380    2580 generic.go:356] "Generic (PLEG): container finished" podID="788a2e1e-f18e-4f17-990d-8cf2916e2bd0" containerID="acb1f3c17a1bc0e89c0bea5d8ccce162552297d03672b9cb5dcc0823397d3132" exitCode=0
Sep 14 02:52:46.034852 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:46.034439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcrlw" event={"ID":"788a2e1e-f18e-4f17-990d-8cf2916e2bd0","Type":"ContainerDied","Data":"acb1f3c17a1bc0e89c0bea5d8ccce162552297d03672b9cb5dcc0823397d3132"}
Sep 14 02:52:46.414504 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:46.414467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:46.414675 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:46.414521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:46.414675 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:46.414607    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:46.414675 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:46.414608    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:46.414675 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:46.414654    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs podName:0da543da-5bad-4aaf-a25b-748ba3cf0a65 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:18.414642594 +0000 UTC m=+65.140600637 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs") pod "network-metrics-daemon-qlq7b" (UID: "0da543da-5bad-4aaf-a25b-748ba3cf0a65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:46.414675 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:46.414668    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret podName:75b4ab6d-419f-495a-b2f5-d62971a327d8 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:18.41466213 +0000 UTC m=+65.140620160 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret") pod "global-pull-secret-syncer-ggtgm" (UID: "75b4ab6d-419f-495a-b2f5-d62971a327d8") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:46.514777 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:46.514753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kggh\" (UniqueName: \"kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh\") pod \"network-check-target-g4jlk\" (UID: \"6e9096c0-240a-460c-a91d-a8a87df60a9a\") " pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:46.514914 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:46.514875    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:46.514914 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:46.514890    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:46.514914 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:46.514899    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8kggh for pod openshift-network-diagnostics/network-check-target-g4jlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:46.515023 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:46.514944    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh podName:6e9096c0-240a-460c-a91d-a8a87df60a9a nodeName:}" failed. No retries permitted until 2026-09-14 02:53:18.514931755 +0000 UTC m=+65.240889798 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kggh" (UniqueName: "kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh") pod "network-check-target-g4jlk" (UID: "6e9096c0-240a-460c-a91d-a8a87df60a9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:46.750854 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:46.750765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:46.751018 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:46.750876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:46.751018 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:46.750888    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4jlk" podUID="6e9096c0-240a-460c-a91d-a8a87df60a9a"
Sep 14 02:52:46.751018 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:46.750892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:46.751018 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:46.750980    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggtgm" podUID="75b4ab6d-419f-495a-b2f5-d62971a327d8"
Sep 14 02:52:46.751243 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:46.751088    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlq7b" podUID="0da543da-5bad-4aaf-a25b-748ba3cf0a65"
Sep 14 02:52:47.579108 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.579081    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeReady"
Sep 14 02:52:47.579559 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.579217    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 14 02:52:47.626183 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.626152    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"]
Sep 14 02:52:47.643688 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.643649    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"]
Sep 14 02:52:47.643857 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.643778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:47.646416 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.646392    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 14 02:52:47.646604 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.646542    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 14 02:52:47.647849 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.647820    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 14 02:52:47.648006 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.647977    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wddxf\""
Sep 14 02:52:47.648160 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.648130    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 14 02:52:47.659575 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.659539    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5b98ff7586-7xqbj"]
Sep 14 02:52:47.659691 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.659653    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.662307 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.662286    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pll7v\""
Sep 14 02:52:47.662484 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.662469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 14 02:52:47.662560 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.662483    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 14 02:52:47.663553 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.663535    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 14 02:52:47.668115 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.668094    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 14 02:52:47.674249 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.674233    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-kmjvv"]
Sep 14 02:52:47.674386 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.674364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.676976 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.676779    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 14 02:52:47.676976 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.676790    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 14 02:52:47.676976 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.676860    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 14 02:52:47.676976 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.676872    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 14 02:52:47.676976 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.676906    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 14 02:52:47.677278 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.677068    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-h4v72\""
Sep 14 02:52:47.677278 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.677252    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 14 02:52:47.696061 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.696041    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"]
Sep 14 02:52:47.696161 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.696147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.701551 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.701527    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 14 02:52:47.701631 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.701609    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 14 02:52:47.701749 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.701733    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mft7c\""
Sep 14 02:52:47.702204 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.702188    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 14 02:52:47.702630 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.702617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 14 02:52:47.707389 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.707373    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 14 02:52:47.716125 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.716109    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"]
Sep 14 02:52:47.716205 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.716132    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"]
Sep 14 02:52:47.716205 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.716145    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b98ff7586-7xqbj"]
Sep 14 02:52:47.716205 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.716158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-kmjvv"]
Sep 14 02:52:47.716205 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.716183    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hzbqh"]
Sep 14 02:52:47.716405 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.716224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:52:47.718395 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.718379    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xzxhw\""
Sep 14 02:52:47.718755 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.718743    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 14 02:52:47.718922 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.718904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 14 02:52:47.719235 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.719220    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 14 02:52:47.725252 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.725234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:47.725325 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.725271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4xl\" (UniqueName: \"kubernetes.io/projected/91e84826-1e48-496a-9617-1bf6cf76c0e4-kube-api-access-kh4xl\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:47.725363 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.725330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/91e84826-1e48-496a-9617-1bf6cf76c0e4-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:47.742126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.742110    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lqmmv"]
Sep 14 02:52:47.742225 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.742211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hzbqh"
Sep 14 02:52:47.744489 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.744470    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gwhfd\""
Sep 14 02:52:47.744578 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.744470    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 14 02:52:47.744578 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.744514    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 14 02:52:47.744743 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.744731    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 14 02:52:47.766447 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.766428    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:47.770112 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.769639    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 14 02:52:47.770112 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.769827    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 14 02:52:47.770410 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.770388    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gj97m\""
Sep 14 02:52:47.771394 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.771377    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"]
Sep 14 02:52:47.771493 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.771420    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hzbqh"]
Sep 14 02:52:47.771493 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.771443    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lqmmv"]
Sep 14 02:52:47.826158 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.826158 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-registry-certificates\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.826158 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lllf\" (UniqueName: \"kubernetes.io/projected/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-kube-api-access-7lllf\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.826382 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.826382 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-image-registry-private-configuration\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.826382 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jczdm\" (UniqueName: \"kubernetes.io/projected/34f503e3-ca4e-4080-9bd6-0e343b5f1b90-kube-api-access-jczdm\") pod \"ingress-canary-hzbqh\" (UID: \"34f503e3-ca4e-4080-9bd6-0e343b5f1b90\") " pod="openshift-ingress-canary/ingress-canary-hzbqh"
Sep 14 02:52:47.826382 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh4jx\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-kube-api-access-hh4jx\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.826382 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34f503e3-ca4e-4080-9bd6-0e343b5f1b90-cert\") pod \"ingress-canary-hzbqh\" (UID: \"34f503e3-ca4e-4080-9bd6-0e343b5f1b90\") " pod="openshift-ingress-canary/ingress-canary-hzbqh"
Sep 14 02:52:47.826382 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-trusted-ca\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.826382 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-stats-auth\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.826382 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-serving-cert\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.826754 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wl4c\" (UniqueName: \"kubernetes.io/projected/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-kube-api-access-7wl4c\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.826754 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c457bc4-98d8-4902-9960-6976161569e1-ca-trust-extracted\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.826754 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-installation-pull-secrets\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.826754 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/91e84826-1e48-496a-9617-1bf6cf76c0e4-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:47.826754 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-default-certificate\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.826754 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.826754 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-snapshots\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.826754 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:47.826754 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:52:47.827182 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-bound-sa-token\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.827182 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k49dw\" (UniqueName: \"kubernetes.io/projected/8caad726-5101-4360-ab79-9267f2b2cd6d-kube-api-access-k49dw\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:52:47.827182 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.827182 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:47.826835    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 02:52:47.827182 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.826893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4xl\" (UniqueName: \"kubernetes.io/projected/91e84826-1e48-496a-9617-1bf6cf76c0e4-kube-api-access-kh4xl\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:47.827182 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:47.826907    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls podName:91e84826-1e48-496a-9617-1bf6cf76c0e4 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:48.326884406 +0000 UTC m=+35.052842452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gcdn" (UID: "91e84826-1e48-496a-9617-1bf6cf76c0e4") : secret "cluster-monitoring-operator-tls" not found
Sep 14 02:52:47.827182 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.827125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-tmp\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.827182 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.827154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-service-ca-bundle\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.827484 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.827355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/91e84826-1e48-496a-9617-1bf6cf76c0e4-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:47.838452 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.838424    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4xl\" (UniqueName: \"kubernetes.io/projected/91e84826-1e48-496a-9617-1bf6cf76c0e4-kube-api-access-kh4xl\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:47.928283 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glzrk\" (UniqueName: \"kubernetes.io/projected/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-kube-api-access-glzrk\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:47.928445 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-default-certificate\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.928445 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.928445 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-snapshots\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.928604 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-bound-sa-token\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.928604 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.928604 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:47.928528    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 14 02:52:47.928604 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:47.928596    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:52:48.428577103 +0000 UTC m=+35.154535150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : secret "router-metrics-certs-default" not found
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:47.928620    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:47.928633    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5: secret "image-registry-tls" not found
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:47.928678    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls podName:2c457bc4-98d8-4902-9960-6976161569e1 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:48.428667245 +0000 UTC m=+35.154625289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls") pod "image-registry-6fcc6d8f6b-qz6n5" (UID: "2c457bc4-98d8-4902-9960-6976161569e1") : secret "image-registry-tls" not found
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-config-volume\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-tmp\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-service-ca-bundle\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lllf\" (UniqueName: \"kubernetes.io/projected/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-kube-api-access-7lllf\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-tmp-dir\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-serving-cert\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wl4c\" (UniqueName: \"kubernetes.io/projected/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-kube-api-access-7wl4c\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.928919 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k49dw\" (UniqueName: \"kubernetes.io/projected/8caad726-5101-4360-ab79-9267f2b2cd6d-kube-api-access-k49dw\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:52:47.929505 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:52:47.929505 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.928999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-image-registry-private-configuration\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.929505 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.929505 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-tmp\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.929505 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-registry-certificates\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.929505 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jczdm\" (UniqueName: \"kubernetes.io/projected/34f503e3-ca4e-4080-9bd6-0e343b5f1b90-kube-api-access-jczdm\") pod \"ingress-canary-hzbqh\" (UID: \"34f503e3-ca4e-4080-9bd6-0e343b5f1b90\") " pod="openshift-ingress-canary/ingress-canary-hzbqh"
Sep 14 02:52:47.929505 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-snapshots\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.929505 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:47.929505 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:47.929145    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 02:52:47.929505 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:47.929199    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls podName:8caad726-5101-4360-ab79-9267f2b2cd6d nodeName:}" failed. No retries permitted until 2026-09-14 02:52:48.429186163 +0000 UTC m=+35.155144195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dthcn" (UID: "8caad726-5101-4360-ab79-9267f2b2cd6d") : secret "samples-operator-tls" not found
Sep 14 02:52:47.929997 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:47.929599    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:52:48.429584972 +0000 UTC m=+35.155543019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : configmap references non-existent config key: service-ca.crt
Sep 14 02:52:47.929997 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh4jx\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-kube-api-access-hh4jx\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.929997 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34f503e3-ca4e-4080-9bd6-0e343b5f1b90-cert\") pod \"ingress-canary-hzbqh\" (UID: \"34f503e3-ca4e-4080-9bd6-0e343b5f1b90\") " pod="openshift-ingress-canary/ingress-canary-hzbqh"
Sep 14 02:52:47.929997 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-trusted-ca\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.929997 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-stats-auth\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.929997 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c457bc4-98d8-4902-9960-6976161569e1-ca-trust-extracted\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.929997 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-registry-certificates\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.929997 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-installation-pull-secrets\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.929997 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.929927    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-service-ca-bundle\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.930747 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.930724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-trusted-ca\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.930923 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.930902    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.931151 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.931126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c457bc4-98d8-4902-9960-6976161569e1-ca-trust-extracted\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.932054 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.932015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-image-registry-private-configuration\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.932210 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.932085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-serving-cert\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.933397 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.933255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-installation-pull-secrets\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.933397 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.933350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-stats-auth\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.934136 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.934116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34f503e3-ca4e-4080-9bd6-0e343b5f1b90-cert\") pod \"ingress-canary-hzbqh\" (UID: \"34f503e3-ca4e-4080-9bd6-0e343b5f1b90\") " pod="openshift-ingress-canary/ingress-canary-hzbqh"
Sep 14 02:52:47.934241 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.934159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-default-certificate\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.941388 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.941368    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8gp2j"]
Sep 14 02:52:47.946697 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.946641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lllf\" (UniqueName: \"kubernetes.io/projected/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-kube-api-access-7lllf\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:47.948663 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.948645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wl4c\" (UniqueName: \"kubernetes.io/projected/1a167d62-c256-4cc5-a3e6-5f6eb19cdb15-kube-api-access-7wl4c\") pod \"insights-operator-5844b856db-kmjvv\" (UID: \"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15\") " pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:47.949203 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.949180    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-bound-sa-token\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.949810 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.949784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k49dw\" (UniqueName: \"kubernetes.io/projected/8caad726-5101-4360-ab79-9267f2b2cd6d-kube-api-access-k49dw\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:52:47.950105 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.950082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jczdm\" (UniqueName: \"kubernetes.io/projected/34f503e3-ca4e-4080-9bd6-0e343b5f1b90-kube-api-access-jczdm\") pod \"ingress-canary-hzbqh\" (UID: \"34f503e3-ca4e-4080-9bd6-0e343b5f1b90\") " pod="openshift-ingress-canary/ingress-canary-hzbqh"
Sep 14 02:52:47.951147 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.951125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh4jx\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-kube-api-access-hh4jx\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:47.973440 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.973410    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8gp2j"]
Sep 14 02:52:47.973596 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.973530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8gp2j"
Sep 14 02:52:47.975886 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.975825    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 14 02:52:47.975992 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.975924    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 14 02:52:47.975992 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:47.975930    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-69c8m\""
Sep 14 02:52:48.003362 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.003339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-kmjvv"
Sep 14 02:52:48.030543 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.030525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glzrk\" (UniqueName: \"kubernetes.io/projected/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-kube-api-access-glzrk\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:48.030622 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.030584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-config-volume\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:48.030622 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.030605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-tmp-dir\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:48.030701 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.030665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:48.030825 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.030813    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 02:52:48.030867 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.030861    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls podName:799a37d7-2bc7-44ff-bd8d-4ddb79b88a38 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:48.530847955 +0000 UTC m=+35.256805985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls") pod "dns-default-lqmmv" (UID: "799a37d7-2bc7-44ff-bd8d-4ddb79b88a38") : secret "dns-default-metrics-tls" not found
Sep 14 02:52:48.031054 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.031014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-tmp-dir\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:48.031144 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.031126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-config-volume\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:48.039072 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.039022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glzrk\" (UniqueName: \"kubernetes.io/projected/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-kube-api-access-glzrk\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:48.049974 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.049951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hzbqh"
Sep 14 02:52:48.131698 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.131656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l22gd\" (UniqueName: \"kubernetes.io/projected/5283b251-dd1e-4ff7-88f4-76b8dd5273ee-kube-api-access-l22gd\") pod \"volume-data-source-validator-86695b7cc7-8gp2j\" (UID: \"5283b251-dd1e-4ff7-88f4-76b8dd5273ee\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8gp2j"
Sep 14 02:52:48.232842 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.232803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22gd\" (UniqueName: \"kubernetes.io/projected/5283b251-dd1e-4ff7-88f4-76b8dd5273ee-kube-api-access-l22gd\") pod \"volume-data-source-validator-86695b7cc7-8gp2j\" (UID: \"5283b251-dd1e-4ff7-88f4-76b8dd5273ee\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8gp2j"
Sep 14 02:52:48.243979 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.243954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l22gd\" (UniqueName: \"kubernetes.io/projected/5283b251-dd1e-4ff7-88f4-76b8dd5273ee-kube-api-access-l22gd\") pod \"volume-data-source-validator-86695b7cc7-8gp2j\" (UID: \"5283b251-dd1e-4ff7-88f4-76b8dd5273ee\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8gp2j"
Sep 14 02:52:48.285953 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.285928    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8gp2j"
Sep 14 02:52:48.334108 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.333903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:48.334108 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.334084    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 02:52:48.334270 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.334163    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls podName:91e84826-1e48-496a-9617-1bf6cf76c0e4 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:49.334144737 +0000 UTC m=+36.060102790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gcdn" (UID: "91e84826-1e48-496a-9617-1bf6cf76c0e4") : secret "cluster-monitoring-operator-tls" not found
Sep 14 02:52:48.370936 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.370689    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-kmjvv"]
Sep 14 02:52:48.375206 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:48.375175    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a167d62_c256_4cc5_a3e6_5f6eb19cdb15.slice/crio-d750a247b4b94120e9c69b91a9b4aebfcaddccde5fa2b6fe83e7abae2d457ffe WatchSource:0}: Error finding container d750a247b4b94120e9c69b91a9b4aebfcaddccde5fa2b6fe83e7abae2d457ffe: Status 404 returned error can't find the container with id d750a247b4b94120e9c69b91a9b4aebfcaddccde5fa2b6fe83e7abae2d457ffe
Sep 14 02:52:48.378800 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.378655    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hzbqh"]
Sep 14 02:52:48.382013 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:48.381990    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f503e3_ca4e_4080_9bd6_0e343b5f1b90.slice/crio-abecc7c4597290bb26b277c50a6181d5945d8c73f182e15976ce6f32450d55d9 WatchSource:0}: Error finding container abecc7c4597290bb26b277c50a6181d5945d8c73f182e15976ce6f32450d55d9: Status 404 returned error can't find the container with id abecc7c4597290bb26b277c50a6181d5945d8c73f182e15976ce6f32450d55d9
Sep 14 02:52:48.435099 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.435056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:52:48.435239 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.435128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:48.435239 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.435185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:48.435239 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.435202    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 02:52:48.435239 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.435235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:48.435436 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.435270    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls podName:8caad726-5101-4360-ab79-9267f2b2cd6d nodeName:}" failed. No retries permitted until 2026-09-14 02:52:49.435250522 +0000 UTC m=+36.161208566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dthcn" (UID: "8caad726-5101-4360-ab79-9267f2b2cd6d") : secret "samples-operator-tls" not found
Sep 14 02:52:48.435436 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.435305    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:52:49.435287427 +0000 UTC m=+36.161245476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : configmap references non-existent config key: service-ca.crt
Sep 14 02:52:48.435436 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.435315    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 02:52:48.435436 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.435327    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5: secret "image-registry-tls" not found
Sep 14 02:52:48.435436 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.435386    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls podName:2c457bc4-98d8-4902-9960-6976161569e1 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:49.435349093 +0000 UTC m=+36.161307140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls") pod "image-registry-6fcc6d8f6b-qz6n5" (UID: "2c457bc4-98d8-4902-9960-6976161569e1") : secret "image-registry-tls" not found
Sep 14 02:52:48.435674 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.435441    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 14 02:52:48.435674 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.435497    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:52:49.435482008 +0000 UTC m=+36.161440054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : secret "router-metrics-certs-default" not found
Sep 14 02:52:48.536165 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.536126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:48.536346 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.536283    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 02:52:48.536346 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:48.536338    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls podName:799a37d7-2bc7-44ff-bd8d-4ddb79b88a38 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:49.53632535 +0000 UTC m=+36.262283386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls") pod "dns-default-lqmmv" (UID: "799a37d7-2bc7-44ff-bd8d-4ddb79b88a38") : secret "dns-default-metrics-tls" not found
Sep 14 02:52:48.582614 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.582582    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8gp2j"]
Sep 14 02:52:48.586894 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:48.586867    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5283b251_dd1e_4ff7_88f4_76b8dd5273ee.slice/crio-25c9345deff8d3abbbf889a8ad259a942f68c04075ed7bf1899858923367dff8 WatchSource:0}: Error finding container 25c9345deff8d3abbbf889a8ad259a942f68c04075ed7bf1899858923367dff8: Status 404 returned error can't find the container with id 25c9345deff8d3abbbf889a8ad259a942f68c04075ed7bf1899858923367dff8
Sep 14 02:52:48.751359 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.751323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:52:48.751359 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.751343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:52:48.751601 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.751343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:52:48.753711 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.753690    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lx47r\""
Sep 14 02:52:48.753824 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.753691    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 02:52:48.753908 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.753691    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hn74x\""
Sep 14 02:52:48.753975 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.753693    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 02:52:48.754141 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.754116    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 02:52:48.754206 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.754155    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 02:52:48.924398 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.924312    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-h97tj"]
Sep 14 02:52:48.940423 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.940397    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-h97tj"]
Sep 14 02:52:48.940569 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.940509    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:48.943307 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.943272    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tdrdb\""
Sep 14 02:52:48.943924 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.943904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 14 02:52:48.944421 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.944066    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 14 02:52:48.944421 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.944225    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 14 02:52:48.944421 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.944310    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 14 02:52:48.949807 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:48.949640    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 14 02:52:49.040857 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.040666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39b540b4-971c-4b5e-9479-1a4a60307be1-serving-cert\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.040857 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.040731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfsk9\" (UniqueName: \"kubernetes.io/projected/39b540b4-971c-4b5e-9479-1a4a60307be1-kube-api-access-jfsk9\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.041129 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.040855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39b540b4-971c-4b5e-9479-1a4a60307be1-trusted-ca\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.041129 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.040920    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39b540b4-971c-4b5e-9479-1a4a60307be1-config\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.045434 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.045401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8gp2j" event={"ID":"5283b251-dd1e-4ff7-88f4-76b8dd5273ee","Type":"ContainerStarted","Data":"25c9345deff8d3abbbf889a8ad259a942f68c04075ed7bf1899858923367dff8"}
Sep 14 02:52:49.047334 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.047291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kmjvv" event={"ID":"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15","Type":"ContainerStarted","Data":"d750a247b4b94120e9c69b91a9b4aebfcaddccde5fa2b6fe83e7abae2d457ffe"}
Sep 14 02:52:49.048931 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.048886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hzbqh" event={"ID":"34f503e3-ca4e-4080-9bd6-0e343b5f1b90","Type":"ContainerStarted","Data":"abecc7c4597290bb26b277c50a6181d5945d8c73f182e15976ce6f32450d55d9"}
Sep 14 02:52:49.141607 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.141565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39b540b4-971c-4b5e-9479-1a4a60307be1-serving-cert\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.141769 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.141640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfsk9\" (UniqueName: \"kubernetes.io/projected/39b540b4-971c-4b5e-9479-1a4a60307be1-kube-api-access-jfsk9\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.141769 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.141722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39b540b4-971c-4b5e-9479-1a4a60307be1-trusted-ca\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.141769 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.141761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39b540b4-971c-4b5e-9479-1a4a60307be1-config\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.142450 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.142422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39b540b4-971c-4b5e-9479-1a4a60307be1-config\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.144077 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.144000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39b540b4-971c-4b5e-9479-1a4a60307be1-trusted-ca\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.152298 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.152275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39b540b4-971c-4b5e-9479-1a4a60307be1-serving-cert\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.155560 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.155485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfsk9\" (UniqueName: \"kubernetes.io/projected/39b540b4-971c-4b5e-9479-1a4a60307be1-kube-api-access-jfsk9\") pod \"console-operator-696884fb9c-h97tj\" (UID: \"39b540b4-971c-4b5e-9479-1a4a60307be1\") " pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.257339 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.257307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:49.344615 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.343927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:49.344615 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.344207    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 02:52:49.344615 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.344271    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls podName:91e84826-1e48-496a-9617-1bf6cf76c0e4 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:51.344252954 +0000 UTC m=+38.070210990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gcdn" (UID: "91e84826-1e48-496a-9617-1bf6cf76c0e4") : secret "cluster-monitoring-operator-tls" not found
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.444637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.444698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.444765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.444859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.444879    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.444942    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:52:51.444922246 +0000 UTC m=+38.170880292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : secret "router-metrics-certs-default" not found
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.444990    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.445004    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5: secret "image-registry-tls" not found
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.445073    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls podName:2c457bc4-98d8-4902-9960-6976161569e1 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:51.445059057 +0000 UTC m=+38.171017103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls") pod "image-registry-6fcc6d8f6b-qz6n5" (UID: "2c457bc4-98d8-4902-9960-6976161569e1") : secret "image-registry-tls" not found
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.445159    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.445197    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls podName:8caad726-5101-4360-ab79-9267f2b2cd6d nodeName:}" failed. No retries permitted until 2026-09-14 02:52:51.445186595 +0000 UTC m=+38.171144640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dthcn" (UID: "8caad726-5101-4360-ab79-9267f2b2cd6d") : secret "samples-operator-tls" not found
Sep 14 02:52:49.445296 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.445259    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:52:51.445249579 +0000 UTC m=+38.171207623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : configmap references non-existent config key: service-ca.crt
Sep 14 02:52:49.546150 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:49.546024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:49.546304 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.546184    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 02:52:49.546304 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:49.546255    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls podName:799a37d7-2bc7-44ff-bd8d-4ddb79b88a38 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:51.546234901 +0000 UTC m=+38.272192944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls") pod "dns-default-lqmmv" (UID: "799a37d7-2bc7-44ff-bd8d-4ddb79b88a38") : secret "dns-default-metrics-tls" not found
Sep 14 02:52:51.362478 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:51.362435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:51.362997 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.362602    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 02:52:51.362997 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.362680    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls podName:91e84826-1e48-496a-9617-1bf6cf76c0e4 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:55.362662825 +0000 UTC m=+42.088620869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gcdn" (UID: "91e84826-1e48-496a-9617-1bf6cf76c0e4") : secret "cluster-monitoring-operator-tls" not found
Sep 14 02:52:51.463405 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:51.463369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:52:51.463538 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:51.463416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:51.463538 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.463512    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 02:52:51.463626 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.463548    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:52:55.463533729 +0000 UTC m=+42.189491764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : configmap references non-existent config key: service-ca.crt
Sep 14 02:52:51.463626 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:51.463572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:51.463733 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:51.463631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:51.463733 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.463652    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 14 02:52:51.463733 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.463673    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls podName:8caad726-5101-4360-ab79-9267f2b2cd6d nodeName:}" failed. No retries permitted until 2026-09-14 02:52:55.463643006 +0000 UTC m=+42.189601050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dthcn" (UID: "8caad726-5101-4360-ab79-9267f2b2cd6d") : secret "samples-operator-tls" not found
Sep 14 02:52:51.463733 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.463706    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 02:52:51.463733 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.463709    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:52:55.463696392 +0000 UTC m=+42.189654423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : secret "router-metrics-certs-default" not found
Sep 14 02:52:51.463733 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.463718    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5: secret "image-registry-tls" not found
Sep 14 02:52:51.463944 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.463762    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls podName:2c457bc4-98d8-4902-9960-6976161569e1 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:55.463750385 +0000 UTC m=+42.189708430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls") pod "image-registry-6fcc6d8f6b-qz6n5" (UID: "2c457bc4-98d8-4902-9960-6976161569e1") : secret "image-registry-tls" not found
Sep 14 02:52:51.564686 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:51.564664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:51.564808 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.564792    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 02:52:51.564851 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:51.564841    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls podName:799a37d7-2bc7-44ff-bd8d-4ddb79b88a38 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:55.564829746 +0000 UTC m=+42.290787792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls") pod "dns-default-lqmmv" (UID: "799a37d7-2bc7-44ff-bd8d-4ddb79b88a38") : secret "dns-default-metrics-tls" not found
Sep 14 02:52:53.525344 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:53.525312    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-h97tj"]
Sep 14 02:52:53.528565 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:52:53.528523    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b540b4_971c_4b5e_9479_1a4a60307be1.slice/crio-a3448ed5a993a9c72167e24f16a3fa46382af8481dad5edb3ca2f747b2c6d766 WatchSource:0}: Error finding container a3448ed5a993a9c72167e24f16a3fa46382af8481dad5edb3ca2f747b2c6d766: Status 404 returned error can't find the container with id a3448ed5a993a9c72167e24f16a3fa46382af8481dad5edb3ca2f747b2c6d766
Sep 14 02:52:54.062276 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.062235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8gp2j" event={"ID":"5283b251-dd1e-4ff7-88f4-76b8dd5273ee","Type":"ContainerStarted","Data":"3b4a7fdab3a962f16a9a6454262eecf6aceaccb80b12c378b24f0bc41bf65e27"}
Sep 14 02:52:54.065763 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.065735    2580 generic.go:356] "Generic (PLEG): container finished" podID="788a2e1e-f18e-4f17-990d-8cf2916e2bd0" containerID="95278f67fc9ac73549db846f2ad1b01782c1cba6a596bac40088b3af36a13856" exitCode=0
Sep 14 02:52:54.065899 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.065806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcrlw" event={"ID":"788a2e1e-f18e-4f17-990d-8cf2916e2bd0","Type":"ContainerDied","Data":"95278f67fc9ac73549db846f2ad1b01782c1cba6a596bac40088b3af36a13856"}
Sep 14 02:52:54.068133 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.068109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kmjvv" event={"ID":"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15","Type":"ContainerStarted","Data":"c5ddecd3498c52b886391ede1ed57d2adcacb7217f31a6cde0db43c25af294e4"}
Sep 14 02:52:54.069681 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.069637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-h97tj" event={"ID":"39b540b4-971c-4b5e-9479-1a4a60307be1","Type":"ContainerStarted","Data":"a3448ed5a993a9c72167e24f16a3fa46382af8481dad5edb3ca2f747b2c6d766"}
Sep 14 02:52:54.071366 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.071339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hzbqh" event={"ID":"34f503e3-ca4e-4080-9bd6-0e343b5f1b90","Type":"ContainerStarted","Data":"e6ef7e43674f0fb691fb4ac887d38113eac8fff23f5bb4816cf261a38ac871ca"}
Sep 14 02:52:54.085402 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.085351    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8gp2j" podStartSLOduration=2.415816518 podStartE2EDuration="7.085335273s" totalImagesPullingTime="4.669518755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:54.08413074 +0000 UTC m=+40.810088792" watchObservedRunningTime="2026-09-14 02:52:54.085335273 +0000 UTC m=+40.811293322"
Sep 14 02:52:54.103287 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.103234    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-kmjvv" podStartSLOduration=5.224067898 podStartE2EDuration="10.103221052s" totalImagesPullingTime="4.879153154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:54.102795087 +0000 UTC m=+40.828753140" watchObservedRunningTime="2026-09-14 02:52:54.103221052 +0000 UTC m=+40.829179105"
Sep 14 02:52:54.125701 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.124800    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hzbqh" podStartSLOduration=2.243108016 podStartE2EDuration="7.124786143s" totalImagesPullingTime="4.881678127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:54.124100134 +0000 UTC m=+40.850058187" watchObservedRunningTime="2026-09-14 02:52:54.124786143 +0000 UTC m=+40.850744196"
Sep 14 02:52:54.530846 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.530814    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hgmx4"]
Sep 14 02:52:54.542069 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.542017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.544242 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.544218    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 14 02:52:54.544409 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.544391    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ztclm\""
Sep 14 02:52:54.544645 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.544621    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 14 02:52:54.550660 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.550638    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hgmx4"]
Sep 14 02:52:54.691749 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.691719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.691749 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.691758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5579be-c3dc-43f1-aa7c-3bd709206921-crio-socket\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.691957 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.691801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5579be-c3dc-43f1-aa7c-3bd709206921-data-volume\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.691957 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.691894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9zwt\" (UniqueName: \"kubernetes.io/projected/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-api-access-q9zwt\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.691957 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.691923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.792506 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.792433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zwt\" (UniqueName: \"kubernetes.io/projected/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-api-access-q9zwt\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.792506 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.792481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.792719 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.792541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.792719 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.792572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5579be-c3dc-43f1-aa7c-3bd709206921-crio-socket\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.792719 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.792619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5579be-c3dc-43f1-aa7c-3bd709206921-data-volume\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.792719 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:54.792645    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 02:52:54.792719 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:54.792694    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 02:52:54.792719 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:54.792715    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm podName:fb5579be-c3dc-43f1-aa7c-3bd709206921 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:55.292694364 +0000 UTC m=+42.018652406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgmx4" (UID: "fb5579be-c3dc-43f1-aa7c-3bd709206921") : configmap "kube-rbac-proxy" not found
Sep 14 02:52:54.792929 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:54.792733    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls podName:fb5579be-c3dc-43f1-aa7c-3bd709206921 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:55.292722226 +0000 UTC m=+42.018680263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgmx4" (UID: "fb5579be-c3dc-43f1-aa7c-3bd709206921") : secret "insights-runtime-extractor-tls" not found
Sep 14 02:52:54.792929 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.792785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5579be-c3dc-43f1-aa7c-3bd709206921-crio-socket\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.792929 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.792903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5579be-c3dc-43f1-aa7c-3bd709206921-data-volume\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:54.803205 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:54.803184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zwt\" (UniqueName: \"kubernetes.io/projected/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-api-access-q9zwt\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:55.077900 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:55.077816    2580 generic.go:356] "Generic (PLEG): container finished" podID="788a2e1e-f18e-4f17-990d-8cf2916e2bd0" containerID="83cfbd6f4902142dc6d933d29356acd427a1d52215ed0442471342d7d7b3e323" exitCode=0
Sep 14 02:52:55.078065 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:55.077896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcrlw" event={"ID":"788a2e1e-f18e-4f17-990d-8cf2916e2bd0","Type":"ContainerDied","Data":"83cfbd6f4902142dc6d933d29356acd427a1d52215ed0442471342d7d7b3e323"}
Sep 14 02:52:55.296256 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:55.296221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:55.296436 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.296374    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 02:52:55.296436 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:55.296388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:55.296546 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.296438    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls podName:fb5579be-c3dc-43f1-aa7c-3bd709206921 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:56.296418903 +0000 UTC m=+43.022376951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgmx4" (UID: "fb5579be-c3dc-43f1-aa7c-3bd709206921") : secret "insights-runtime-extractor-tls" not found
Sep 14 02:52:55.296546 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.296501    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 02:52:55.296646 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.296568    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm podName:fb5579be-c3dc-43f1-aa7c-3bd709206921 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:56.296554215 +0000 UTC m=+43.022512261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgmx4" (UID: "fb5579be-c3dc-43f1-aa7c-3bd709206921") : configmap "kube-rbac-proxy" not found
Sep 14 02:52:55.397238 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:55.397154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:52:55.397393 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.397315    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 02:52:55.397393 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.397385    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls podName:91e84826-1e48-496a-9617-1bf6cf76c0e4 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:03.397367497 +0000 UTC m=+50.123325540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gcdn" (UID: "91e84826-1e48-496a-9617-1bf6cf76c0e4") : secret "cluster-monitoring-operator-tls" not found
Sep 14 02:52:55.497790 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:55.497757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:52:55.497923 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:55.497801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:55.497923 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:55.497851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:52:55.497923 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:55.497897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:52:55.498056 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.497896    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 02:52:55.498056 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.497990    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 02:52:55.498056 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.497997    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:53:03.497978881 +0000 UTC m=+50.223936925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : configmap references non-existent config key: service-ca.crt
Sep 14 02:52:55.498056 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.498004    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5: secret "image-registry-tls" not found
Sep 14 02:52:55.498056 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.498019    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls podName:8caad726-5101-4360-ab79-9267f2b2cd6d nodeName:}" failed. No retries permitted until 2026-09-14 02:53:03.498009799 +0000 UTC m=+50.223967831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dthcn" (UID: "8caad726-5101-4360-ab79-9267f2b2cd6d") : secret "samples-operator-tls" not found
Sep 14 02:52:55.498056 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.497942    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 14 02:52:55.498299 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.498068    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls podName:2c457bc4-98d8-4902-9960-6976161569e1 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:03.498054156 +0000 UTC m=+50.224012201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls") pod "image-registry-6fcc6d8f6b-qz6n5" (UID: "2c457bc4-98d8-4902-9960-6976161569e1") : secret "image-registry-tls" not found
Sep 14 02:52:55.498299 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.498082    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:53:03.498075683 +0000 UTC m=+50.224033713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : secret "router-metrics-certs-default" not found
Sep 14 02:52:55.598555 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:55.598530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:52:55.598886 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.598743    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 02:52:55.598886 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:55.598821    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls podName:799a37d7-2bc7-44ff-bd8d-4ddb79b88a38 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:03.598799358 +0000 UTC m=+50.324757412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls") pod "dns-default-lqmmv" (UID: "799a37d7-2bc7-44ff-bd8d-4ddb79b88a38") : secret "dns-default-metrics-tls" not found
Sep 14 02:52:55.891118 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:55.891099    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6ps5c_c3944957-e448-4279-834a-1b091a14afe8/dns-node-resolver/0.log"
Sep 14 02:52:56.081696 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:56.081675    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/0.log"
Sep 14 02:52:56.081773 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:56.081710    2580 generic.go:356] "Generic (PLEG): container finished" podID="39b540b4-971c-4b5e-9479-1a4a60307be1" containerID="d40e31a4e6cb96c602c27c84189493ef9e2dae749c33d80f263858d7271ee762" exitCode=255
Sep 14 02:52:56.081819 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:56.081801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-h97tj" event={"ID":"39b540b4-971c-4b5e-9479-1a4a60307be1","Type":"ContainerDied","Data":"d40e31a4e6cb96c602c27c84189493ef9e2dae749c33d80f263858d7271ee762"}
Sep 14 02:52:56.081984 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:56.081960    2580 scope.go:117] "RemoveContainer" containerID="d40e31a4e6cb96c602c27c84189493ef9e2dae749c33d80f263858d7271ee762"
Sep 14 02:52:56.085644 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:56.085625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcrlw" event={"ID":"788a2e1e-f18e-4f17-990d-8cf2916e2bd0","Type":"ContainerStarted","Data":"cbecfba9fce6df5897e3e37b13a4bdf5e78dfa64c37595b9fecc02a0a742521d"}
Sep 14 02:52:56.122414 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:56.122375    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vcrlw" podStartSLOduration=8.168560867 podStartE2EDuration="42.122363999s" totalImagesPullingTime="33.953803132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:56.121395239 +0000 UTC m=+42.847353313" watchObservedRunningTime="2026-09-14 02:52:56.122363999 +0000 UTC m=+42.848322072"
Sep 14 02:52:56.304051 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:56.303989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:56.304184 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:56.304148    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 02:52:56.304246 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:56.304193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:56.304246 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:56.304214    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls podName:fb5579be-c3dc-43f1-aa7c-3bd709206921 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:58.304196564 +0000 UTC m=+45.030154594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgmx4" (UID: "fb5579be-c3dc-43f1-aa7c-3bd709206921") : secret "insights-runtime-extractor-tls" not found
Sep 14 02:52:56.304357 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:56.304257    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 02:52:56.304357 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:56.304313    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm podName:fb5579be-c3dc-43f1-aa7c-3bd709206921 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:58.304295641 +0000 UTC m=+45.030253674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgmx4" (UID: "fb5579be-c3dc-43f1-aa7c-3bd709206921") : configmap "kube-rbac-proxy" not found
Sep 14 02:52:56.690733 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:56.690710    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-852l7_6d2fa0b7-cc69-4546-b798-f5114afde64a/node-ca/0.log"
Sep 14 02:52:57.089268 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:57.089245    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 02:52:57.089635 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:57.089620    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/0.log"
Sep 14 02:52:57.089694 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:57.089654    2580 generic.go:356] "Generic (PLEG): container finished" podID="39b540b4-971c-4b5e-9479-1a4a60307be1" containerID="7d8b44e5d6654df1964de0cbc8474e9793d95cebb66ade4207ccd2dc8c7bbca1" exitCode=255
Sep 14 02:52:57.089725 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:57.089685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-h97tj" event={"ID":"39b540b4-971c-4b5e-9479-1a4a60307be1","Type":"ContainerDied","Data":"7d8b44e5d6654df1964de0cbc8474e9793d95cebb66ade4207ccd2dc8c7bbca1"}
Sep 14 02:52:57.102577 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:57.102554    2580 scope.go:117] "RemoveContainer" containerID="d40e31a4e6cb96c602c27c84189493ef9e2dae749c33d80f263858d7271ee762"
Sep 14 02:52:57.102722 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:57.102662    2580 scope.go:117] "RemoveContainer" containerID="7d8b44e5d6654df1964de0cbc8474e9793d95cebb66ade4207ccd2dc8c7bbca1"
Sep 14 02:52:57.102843 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:57.102826    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-h97tj_openshift-console-operator(39b540b4-971c-4b5e-9479-1a4a60307be1)\"" pod="openshift-console-operator/console-operator-696884fb9c-h97tj" podUID="39b540b4-971c-4b5e-9479-1a4a60307be1"
Sep 14 02:52:57.698528 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:57.698507    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hzbqh_34f503e3-ca4e-4080-9bd6-0e343b5f1b90/serve-healthcheck-canary/0.log"
Sep 14 02:52:58.094279 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:58.094252    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 02:52:58.094588 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:58.094573    2580 scope.go:117] "RemoveContainer" containerID="7d8b44e5d6654df1964de0cbc8474e9793d95cebb66ade4207ccd2dc8c7bbca1"
Sep 14 02:52:58.094734 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:58.094716    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-h97tj_openshift-console-operator(39b540b4-971c-4b5e-9479-1a4a60307be1)\"" pod="openshift-console-operator/console-operator-696884fb9c-h97tj" podUID="39b540b4-971c-4b5e-9479-1a4a60307be1"
Sep 14 02:52:58.320626 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:58.320593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:58.320782 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:58.320651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:52:58.320782 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:58.320714    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 02:52:58.320782 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:58.320773    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm podName:fb5579be-c3dc-43f1-aa7c-3bd709206921 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:02.320758308 +0000 UTC m=+49.046716342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgmx4" (UID: "fb5579be-c3dc-43f1-aa7c-3bd709206921") : configmap "kube-rbac-proxy" not found
Sep 14 02:52:58.320943 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:58.320834    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 02:52:58.320943 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:58.320897    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls podName:fb5579be-c3dc-43f1-aa7c-3bd709206921 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:02.320879681 +0000 UTC m=+49.046837731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgmx4" (UID: "fb5579be-c3dc-43f1-aa7c-3bd709206921") : secret "insights-runtime-extractor-tls" not found
Sep 14 02:52:59.258046 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:59.258003    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:59.258407 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:59.258092    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:52:59.258407 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:52:59.258384    2580 scope.go:117] "RemoveContainer" containerID="7d8b44e5d6654df1964de0cbc8474e9793d95cebb66ade4207ccd2dc8c7bbca1"
Sep 14 02:52:59.258540 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:52:59.258521    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-h97tj_openshift-console-operator(39b540b4-971c-4b5e-9479-1a4a60307be1)\"" pod="openshift-console-operator/console-operator-696884fb9c-h97tj" podUID="39b540b4-971c-4b5e-9479-1a4a60307be1"
Sep 14 02:53:02.352801 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:02.352765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:53:02.353170 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:02.352826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:53:02.353170 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:02.352883    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 02:53:02.353170 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:02.352915    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 02:53:02.353170 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:02.352948    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm podName:fb5579be-c3dc-43f1-aa7c-3bd709206921 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:10.352932196 +0000 UTC m=+57.078890226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgmx4" (UID: "fb5579be-c3dc-43f1-aa7c-3bd709206921") : configmap "kube-rbac-proxy" not found
Sep 14 02:53:02.353170 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:02.352962    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls podName:fb5579be-c3dc-43f1-aa7c-3bd709206921 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:10.352956239 +0000 UTC m=+57.078914268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgmx4" (UID: "fb5579be-c3dc-43f1-aa7c-3bd709206921") : secret "insights-runtime-extractor-tls" not found
Sep 14 02:53:03.464280 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:03.464228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:53:03.464791 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.464393    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 02:53:03.464791 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.464482    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls podName:91e84826-1e48-496a-9617-1bf6cf76c0e4 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:19.464461336 +0000 UTC m=+66.190419371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gcdn" (UID: "91e84826-1e48-496a-9617-1bf6cf76c0e4") : secret "cluster-monitoring-operator-tls" not found
Sep 14 02:53:03.565080 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:03.565025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:53:03.565239 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:03.565091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:53:03.565239 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:03.565153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:53:03.565239 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.565174    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 02:53:03.565239 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:03.565197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:53:03.565239 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.565232    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls podName:8caad726-5101-4360-ab79-9267f2b2cd6d nodeName:}" failed. No retries permitted until 2026-09-14 02:53:19.565213757 +0000 UTC m=+66.291171791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dthcn" (UID: "8caad726-5101-4360-ab79-9267f2b2cd6d") : secret "samples-operator-tls" not found
Sep 14 02:53:03.565417 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.565257    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:53:19.565244376 +0000 UTC m=+66.291202412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : configmap references non-existent config key: service-ca.crt
Sep 14 02:53:03.565417 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.565260    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 14 02:53:03.565417 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.565290    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs podName:4a5a2f9a-1382-4c95-a2d7-ab0484564a3c nodeName:}" failed. No retries permitted until 2026-09-14 02:53:19.565280456 +0000 UTC m=+66.291238487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs") pod "router-default-5b98ff7586-7xqbj" (UID: "4a5a2f9a-1382-4c95-a2d7-ab0484564a3c") : secret "router-metrics-certs-default" not found
Sep 14 02:53:03.565417 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.565304    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 02:53:03.565417 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.565321    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5: secret "image-registry-tls" not found
Sep 14 02:53:03.565417 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.565368    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls podName:2c457bc4-98d8-4902-9960-6976161569e1 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:19.565357211 +0000 UTC m=+66.291315263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls") pod "image-registry-6fcc6d8f6b-qz6n5" (UID: "2c457bc4-98d8-4902-9960-6976161569e1") : secret "image-registry-tls" not found
Sep 14 02:53:03.666234 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:03.666153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:53:03.666373 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.666304    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 02:53:03.666373 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:03.666371    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls podName:799a37d7-2bc7-44ff-bd8d-4ddb79b88a38 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:19.666355621 +0000 UTC m=+66.392313658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls") pod "dns-default-lqmmv" (UID: "799a37d7-2bc7-44ff-bd8d-4ddb79b88a38") : secret "dns-default-metrics-tls" not found
Sep 14 02:53:10.422921 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:10.422877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:53:10.423300 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:10.422992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:53:10.423300 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:10.423049    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 02:53:10.423300 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:53:10.423118    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls podName:fb5579be-c3dc-43f1-aa7c-3bd709206921 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:26.423100638 +0000 UTC m=+73.149058672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgmx4" (UID: "fb5579be-c3dc-43f1-aa7c-3bd709206921") : secret "insights-runtime-extractor-tls" not found
Sep 14 02:53:10.423523 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:10.423507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5579be-c3dc-43f1-aa7c-3bd709206921-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:53:11.751289 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:11.751256    2580 scope.go:117] "RemoveContainer" containerID="7d8b44e5d6654df1964de0cbc8474e9793d95cebb66ade4207ccd2dc8c7bbca1"
Sep 14 02:53:12.130304 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:12.130273    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 02:53:12.130449 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:12.130366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-h97tj" event={"ID":"39b540b4-971c-4b5e-9479-1a4a60307be1","Type":"ContainerStarted","Data":"25f81b1b3c23d46aea471c47ac6f65aab11efd73de8dd223e60193d0a77f0232"}
Sep 14 02:53:12.130706 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:12.130677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:53:12.141339 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:12.141317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-h97tj"
Sep 14 02:53:12.152306 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:12.152252    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-h97tj" podStartSLOduration=21.793503251 podStartE2EDuration="24.15223333s" totalImagesPullingTime="2.358730079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:12.150599779 +0000 UTC m=+58.876557831" watchObservedRunningTime="2026-09-14 02:53:12.15223333 +0000 UTC m=+58.878191384"
Sep 14 02:53:14.042628 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:14.042599    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq4zd"
Sep 14 02:53:18.492276 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.492241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:53:18.492658 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.492366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:53:18.494721 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.494701    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 02:53:18.494835 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.494736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 02:53:18.505453 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.505431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b4ab6d-419f-495a-b2f5-d62971a327d8-original-pull-secret\") pod \"global-pull-secret-syncer-ggtgm\" (UID: \"75b4ab6d-419f-495a-b2f5-d62971a327d8\") " pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:53:18.505567 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.505550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da543da-5bad-4aaf-a25b-748ba3cf0a65-metrics-certs\") pod \"network-metrics-daemon-qlq7b\" (UID: \"0da543da-5bad-4aaf-a25b-748ba3cf0a65\") " pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:53:18.593518 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.593488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kggh\" (UniqueName: \"kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh\") pod \"network-check-target-g4jlk\" (UID: \"6e9096c0-240a-460c-a91d-a8a87df60a9a\") " pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:53:18.595941 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.595923    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 02:53:18.606558 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.606544    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 02:53:18.617600 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.617579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kggh\" (UniqueName: \"kubernetes.io/projected/6e9096c0-240a-460c-a91d-a8a87df60a9a-kube-api-access-8kggh\") pod \"network-check-target-g4jlk\" (UID: \"6e9096c0-240a-460c-a91d-a8a87df60a9a\") " pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:53:18.765152 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.765129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hn74x\""
Sep 14 02:53:18.772741 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.772716    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lx47r\""
Sep 14 02:53:18.773700 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.773685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:53:18.776204 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.776190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggtgm"
Sep 14 02:53:18.781651 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:18.781635    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlq7b"
Sep 14 02:53:19.063743 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.063687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g4jlk"]
Sep 14 02:53:19.065183 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.065120    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ggtgm"]
Sep 14 02:53:19.066520 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:19.066495    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9096c0_240a_460c_a91d_a8a87df60a9a.slice/crio-456d9e95f66ea2b77084bcf08d7cdebba3a1202cac6dc50b8bdb00213b935ae1 WatchSource:0}: Error finding container 456d9e95f66ea2b77084bcf08d7cdebba3a1202cac6dc50b8bdb00213b935ae1: Status 404 returned error can't find the container with id 456d9e95f66ea2b77084bcf08d7cdebba3a1202cac6dc50b8bdb00213b935ae1
Sep 14 02:53:19.067334 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:19.067312    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b4ab6d_419f_495a_b2f5_d62971a327d8.slice/crio-c8be016c19912b48971e8a11fead6497b3d5f8aa61c2e00c4d9fd65d6f8b56a9 WatchSource:0}: Error finding container c8be016c19912b48971e8a11fead6497b3d5f8aa61c2e00c4d9fd65d6f8b56a9: Status 404 returned error can't find the container with id c8be016c19912b48971e8a11fead6497b3d5f8aa61c2e00c4d9fd65d6f8b56a9
Sep 14 02:53:19.105832 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.105813    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qlq7b"]
Sep 14 02:53:19.106772 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:19.106756    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da543da_5bad_4aaf_a25b_748ba3cf0a65.slice/crio-69348cb7bf7e71c44e34209f2a06a39d562680fa99c7dc1be749971acc3bc156 WatchSource:0}: Error finding container 69348cb7bf7e71c44e34209f2a06a39d562680fa99c7dc1be749971acc3bc156: Status 404 returned error can't find the container with id 69348cb7bf7e71c44e34209f2a06a39d562680fa99c7dc1be749971acc3bc156
Sep 14 02:53:19.148231 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.148200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g4jlk" event={"ID":"6e9096c0-240a-460c-a91d-a8a87df60a9a","Type":"ContainerStarted","Data":"456d9e95f66ea2b77084bcf08d7cdebba3a1202cac6dc50b8bdb00213b935ae1"}
Sep 14 02:53:19.149329 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.149307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ggtgm" event={"ID":"75b4ab6d-419f-495a-b2f5-d62971a327d8","Type":"ContainerStarted","Data":"c8be016c19912b48971e8a11fead6497b3d5f8aa61c2e00c4d9fd65d6f8b56a9"}
Sep 14 02:53:19.150288 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.150271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlq7b" event={"ID":"0da543da-5bad-4aaf-a25b-748ba3cf0a65","Type":"ContainerStarted","Data":"69348cb7bf7e71c44e34209f2a06a39d562680fa99c7dc1be749971acc3bc156"}
Sep 14 02:53:19.502459 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.502385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:53:19.504632 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.504612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e84826-1e48-496a-9617-1bf6cf76c0e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gcdn\" (UID: \"91e84826-1e48-496a-9617-1bf6cf76c0e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:53:19.603247 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.603213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:53:19.603393 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.603270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:53:19.603393 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.603316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:53:19.603393 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.603345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:53:19.604101 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.604076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-service-ca-bundle\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:53:19.605965 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.605939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5a2f9a-1382-4c95-a2d7-ab0484564a3c-metrics-certs\") pod \"router-default-5b98ff7586-7xqbj\" (UID: \"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c\") " pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:53:19.606095 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.606004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls\") pod \"image-registry-6fcc6d8f6b-qz6n5\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") " pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:53:19.606181 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.606155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8caad726-5101-4360-ab79-9267f2b2cd6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dthcn\" (UID: \"8caad726-5101-4360-ab79-9267f2b2cd6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:53:19.704419 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.704388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:53:19.707899 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.707873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799a37d7-2bc7-44ff-bd8d-4ddb79b88a38-metrics-tls\") pod \"dns-default-lqmmv\" (UID: \"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38\") " pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:53:19.757909 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.757883    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wddxf\""
Sep 14 02:53:19.765937 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.765911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"
Sep 14 02:53:19.772414 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.772389    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pll7v\""
Sep 14 02:53:19.781264 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.781243    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:53:19.785179 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.785159    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-h4v72\""
Sep 14 02:53:19.794175 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.794111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:53:19.826877 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.826676    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xzxhw\""
Sep 14 02:53:19.835286 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.835262    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"
Sep 14 02:53:19.886703 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.886372    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gj97m\""
Sep 14 02:53:19.895610 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:19.895499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:53:20.054314 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:20.054230    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn"]
Sep 14 02:53:20.060615 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:20.060585    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e84826_1e48_496a_9617_1bf6cf76c0e4.slice/crio-a4eb3e44a0e5355179369ff5666d61c0625ad173073e5acb9669028b94fc0424 WatchSource:0}: Error finding container a4eb3e44a0e5355179369ff5666d61c0625ad173073e5acb9669028b94fc0424: Status 404 returned error can't find the container with id a4eb3e44a0e5355179369ff5666d61c0625ad173073e5acb9669028b94fc0424
Sep 14 02:53:20.096364 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:20.096335    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b98ff7586-7xqbj"]
Sep 14 02:53:20.098877 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:20.098857    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"]
Sep 14 02:53:20.100987 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:20.100817    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5a2f9a_1382_4c95_a2d7_ab0484564a3c.slice/crio-33878f16a4c02d71e4b3fdc80e85cc28565f52e8fa5c15ae6666c5a12af9da61 WatchSource:0}: Error finding container 33878f16a4c02d71e4b3fdc80e85cc28565f52e8fa5c15ae6666c5a12af9da61: Status 404 returned error can't find the container with id 33878f16a4c02d71e4b3fdc80e85cc28565f52e8fa5c15ae6666c5a12af9da61
Sep 14 02:53:20.102116 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:20.102078    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c457bc4_98d8_4902_9960_6976161569e1.slice/crio-696ff03025278d20be8b2ef65376e75043df56c3199a26edacd59b67b30bc992 WatchSource:0}: Error finding container 696ff03025278d20be8b2ef65376e75043df56c3199a26edacd59b67b30bc992: Status 404 returned error can't find the container with id 696ff03025278d20be8b2ef65376e75043df56c3199a26edacd59b67b30bc992
Sep 14 02:53:20.155076 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:20.154975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn" event={"ID":"91e84826-1e48-496a-9617-1bf6cf76c0e4","Type":"ContainerStarted","Data":"a4eb3e44a0e5355179369ff5666d61c0625ad173073e5acb9669028b94fc0424"}
Sep 14 02:53:20.156916 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:20.156849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b98ff7586-7xqbj" event={"ID":"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c","Type":"ContainerStarted","Data":"33878f16a4c02d71e4b3fdc80e85cc28565f52e8fa5c15ae6666c5a12af9da61"}
Sep 14 02:53:20.159893 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:20.159858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5" event={"ID":"2c457bc4-98d8-4902-9960-6976161569e1","Type":"ContainerStarted","Data":"696ff03025278d20be8b2ef65376e75043df56c3199a26edacd59b67b30bc992"}
Sep 14 02:53:20.166640 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:20.166567    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn"]
Sep 14 02:53:20.205361 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:20.205292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lqmmv"]
Sep 14 02:53:20.392603 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:20.392569    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799a37d7_2bc7_44ff_bd8d_4ddb79b88a38.slice/crio-e1a44ea750d051f65d79e70d825664531ffe118a08e21f2c0ef21abd2cb06990 WatchSource:0}: Error finding container e1a44ea750d051f65d79e70d825664531ffe118a08e21f2c0ef21abd2cb06990: Status 404 returned error can't find the container with id e1a44ea750d051f65d79e70d825664531ffe118a08e21f2c0ef21abd2cb06990
Sep 14 02:53:21.178107 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.177634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlq7b" event={"ID":"0da543da-5bad-4aaf-a25b-748ba3cf0a65","Type":"ContainerStarted","Data":"75350e0b1116818b07544369484021c83d4157b3e2af88370fe9f31b27ef0d44"}
Sep 14 02:53:21.178107 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.177728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlq7b" event={"ID":"0da543da-5bad-4aaf-a25b-748ba3cf0a65","Type":"ContainerStarted","Data":"e89f177f74f5c90dbb1a6725e312980fa3b13cb1e3aa5363317b9c13fae75ff8"}
Sep 14 02:53:21.181587 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.181554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b98ff7586-7xqbj" event={"ID":"4a5a2f9a-1382-4c95-a2d7-ab0484564a3c","Type":"ContainerStarted","Data":"b76ef5d5cc63827d3ccbc915cc1ef97791559cfc945adfbd682cb1ecbd6674d2"}
Sep 14 02:53:21.184716 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.184197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5" event={"ID":"2c457bc4-98d8-4902-9960-6976161569e1","Type":"ContainerStarted","Data":"32fa618c4c731a5f38b5c5c3335fd21e0f12751525e5c417ac6df328b186c63b"}
Sep 14 02:53:21.184716 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.184624    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:53:21.186645 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.186590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqmmv" event={"ID":"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38","Type":"ContainerStarted","Data":"e1a44ea750d051f65d79e70d825664531ffe118a08e21f2c0ef21abd2cb06990"}
Sep 14 02:53:21.188853 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.188803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn" event={"ID":"8caad726-5101-4360-ab79-9267f2b2cd6d","Type":"ContainerStarted","Data":"28f9df8b4b8a15fd13a66d7c88874c785f8921706169cb2a477075abe7c0931f"}
Sep 14 02:53:21.195165 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.195118    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qlq7b" podStartSLOduration=65.873826205 podStartE2EDuration="1m7.195103103s" totalImagesPullingTime="1.321276898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:21.194427563 +0000 UTC m=+67.920385616" watchObservedRunningTime="2026-09-14 02:53:21.195103103 +0000 UTC m=+67.921061157"
Sep 14 02:53:21.216714 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.216666    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5" podStartSLOduration=67.21664939 podStartE2EDuration="1m7.21664939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:21.215074155 +0000 UTC m=+67.941032208" watchObservedRunningTime="2026-09-14 02:53:21.21664939 +0000 UTC m=+67.942607443"
Sep 14 02:53:21.235835 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.234345    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5b98ff7586-7xqbj" podStartSLOduration=37.234328721 podStartE2EDuration="37.234328721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:21.234101074 +0000 UTC m=+67.960059129" watchObservedRunningTime="2026-09-14 02:53:21.234328721 +0000 UTC m=+67.960286775"
Sep 14 02:53:21.795158 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.794518    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:53:21.801364 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:21.801162    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:53:22.193668 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:22.193466    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:53:22.194649 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:22.194626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5b98ff7586-7xqbj"
Sep 14 02:53:26.463866 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:26.463834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:53:26.466640 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:26.466608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5579be-c3dc-43f1-aa7c-3bd709206921-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgmx4\" (UID: \"fb5579be-c3dc-43f1-aa7c-3bd709206921\") " pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:53:26.654922 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:26.654887    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ztclm\""
Sep 14 02:53:26.663635 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:26.663567    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hgmx4"
Sep 14 02:53:26.944973 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:26.944867    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hgmx4"]
Sep 14 02:53:26.949319 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:26.949289    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5579be_c3dc_43f1_aa7c_3bd709206921.slice/crio-f9b3e37b8bcbe4374a9f9f535c7c815976d88cbcaed1ed44a2b77d2fe6da5b14 WatchSource:0}: Error finding container f9b3e37b8bcbe4374a9f9f535c7c815976d88cbcaed1ed44a2b77d2fe6da5b14: Status 404 returned error can't find the container with id f9b3e37b8bcbe4374a9f9f535c7c815976d88cbcaed1ed44a2b77d2fe6da5b14
Sep 14 02:53:27.213355 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.213280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g4jlk" event={"ID":"6e9096c0-240a-460c-a91d-a8a87df60a9a","Type":"ContainerStarted","Data":"3745f92033cdcf1d5b001c628dd96b9e42fa6ebfa59dfa803440d10c20e00302"}
Sep 14 02:53:27.213494 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.213410    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:53:27.215084 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.215051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn" event={"ID":"91e84826-1e48-496a-9617-1bf6cf76c0e4","Type":"ContainerStarted","Data":"20204e6de1e51aeae6e9727f6672c8bfb57353401a0d142dfbe59a087d7cacf8"}
Sep 14 02:53:27.217010 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.216985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqmmv" event={"ID":"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38","Type":"ContainerStarted","Data":"557091326a05a86777609f1d18d0a37f7892875e0bd13f07f35a178b1cd9a121"}
Sep 14 02:53:27.217128 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.217048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqmmv" event={"ID":"799a37d7-2bc7-44ff-bd8d-4ddb79b88a38","Type":"ContainerStarted","Data":"191af18250508ffc569fe89f908c1e068b3fcb33895fed94e7ec174c5efbe824"}
Sep 14 02:53:27.217128 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.217090    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:53:27.218928 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.218901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn" event={"ID":"8caad726-5101-4360-ab79-9267f2b2cd6d","Type":"ContainerStarted","Data":"6bcf6ec993d8f9154ab7e5c668d190003091e0b8d7d28db3dd9e6db688a7c974"}
Sep 14 02:53:27.219015 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.218958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn" event={"ID":"8caad726-5101-4360-ab79-9267f2b2cd6d","Type":"ContainerStarted","Data":"b9f8ba5e4e23d8935edbdf6aa95472f4f9d66d9325a92271099b5105a289b1b0"}
Sep 14 02:53:27.220385 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.220354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgmx4" event={"ID":"fb5579be-c3dc-43f1-aa7c-3bd709206921","Type":"ContainerStarted","Data":"bdc3b33ce96628d4925fda3e555f4b1e01a46b26db4e2af778312fdc3febcb05"}
Sep 14 02:53:27.220469 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.220413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgmx4" event={"ID":"fb5579be-c3dc-43f1-aa7c-3bd709206921","Type":"ContainerStarted","Data":"f9b3e37b8bcbe4374a9f9f535c7c815976d88cbcaed1ed44a2b77d2fe6da5b14"}
Sep 14 02:53:27.221752 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.221734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ggtgm" event={"ID":"75b4ab6d-419f-495a-b2f5-d62971a327d8","Type":"ContainerStarted","Data":"cdfa0a04fe463b8a69a39c2fb059318491dbf290e39ceede4b0ecbf0bb36ffe8"}
Sep 14 02:53:27.229635 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.229597    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g4jlk" podStartSLOduration=65.872060504 podStartE2EDuration="1m13.229584408s" totalImagesPullingTime="7.357523904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:27.228373217 +0000 UTC m=+73.954331268" watchObservedRunningTime="2026-09-14 02:53:27.229584408 +0000 UTC m=+73.955542459"
Sep 14 02:53:27.246025 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.245991    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dthcn" podStartSLOduration=35.25853237 podStartE2EDuration="41.245978357s" totalImagesPullingTime="5.987445987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:27.245384965 +0000 UTC m=+73.971343018" watchObservedRunningTime="2026-09-14 02:53:27.245978357 +0000 UTC m=+73.971936409"
Sep 14 02:53:27.265020 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.264984    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gcdn" podStartSLOduration=36.907711215 podStartE2EDuration="43.264971965s" totalImagesPullingTime="6.35726075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:27.264053913 +0000 UTC m=+73.990011957" watchObservedRunningTime="2026-09-14 02:53:27.264971965 +0000 UTC m=+73.990930016"
Sep 14 02:53:27.279725 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.279690    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lqmmv" podStartSLOduration=34.254940487 podStartE2EDuration="40.27967796s" totalImagesPullingTime="6.024737473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:27.27875411 +0000 UTC m=+74.004712161" watchObservedRunningTime="2026-09-14 02:53:27.27967796 +0000 UTC m=+74.005636012"
Sep 14 02:53:27.293864 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:27.293766    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ggtgm" podStartSLOduration=65.921362495 podStartE2EDuration="1m13.2937517s" totalImagesPullingTime="7.372389205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:27.292933784 +0000 UTC m=+74.018891838" watchObservedRunningTime="2026-09-14 02:53:27.2937517 +0000 UTC m=+74.019709755"
Sep 14 02:53:28.226878 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:28.226838    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgmx4" event={"ID":"fb5579be-c3dc-43f1-aa7c-3bd709206921","Type":"ContainerStarted","Data":"99c1e677ddfae1fd2f4f441fc379618bf787b5f542005341282fb3b81edbfbfe"}
Sep 14 02:53:29.726080 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.726043    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"]
Sep 14 02:53:29.728510 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.728486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:29.731565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.731531    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 14 02:53:29.731698 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.731531    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 14 02:53:29.732431 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.732404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 14 02:53:29.732431 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.732417    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 14 02:53:29.748538 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.748512    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"]
Sep 14 02:53:29.749721 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.749698    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8"]
Sep 14 02:53:29.751721 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.751699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8"
Sep 14 02:53:29.754268 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.754045    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-znp8s\""
Sep 14 02:53:29.754268 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.754116    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 14 02:53:29.763911 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.763878    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8"]
Sep 14 02:53:29.788290 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.788271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e1355a1-2166-47d0-a829-923ef50e7c7a-tmp\") pod \"klusterlet-addon-workmgr-586c8749df-vtb4g\" (UID: \"6e1355a1-2166-47d0-a829-923ef50e7c7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:29.788379 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.788306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6e1355a1-2166-47d0-a829-923ef50e7c7a-klusterlet-config\") pod \"klusterlet-addon-workmgr-586c8749df-vtb4g\" (UID: \"6e1355a1-2166-47d0-a829-923ef50e7c7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:29.788424 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.788405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2sgz\" (UniqueName: \"kubernetes.io/projected/6e1355a1-2166-47d0-a829-923ef50e7c7a-kube-api-access-v2sgz\") pod \"klusterlet-addon-workmgr-586c8749df-vtb4g\" (UID: \"6e1355a1-2166-47d0-a829-923ef50e7c7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:29.795528 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.795503    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-565bccd9d-thp2r"]
Sep 14 02:53:29.797464 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.797446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.799206 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.799186    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 14 02:53:29.799510 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.799489    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 14 02:53:29.799635 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.799559    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xzm62\""
Sep 14 02:53:29.799848 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.799827    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 14 02:53:29.799848 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.799842    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 14 02:53:29.799962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.799894    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 14 02:53:29.800178 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.800163    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 14 02:53:29.800365 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.800349    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-p7bpw"]
Sep 14 02:53:29.801642 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.801624    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 14 02:53:29.802065 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.802048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-p7bpw"
Sep 14 02:53:29.805639 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.805618    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zbm55\""
Sep 14 02:53:29.817406 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.816943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565bccd9d-thp2r"]
Sep 14 02:53:29.872214 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.872189    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-p7bpw"]
Sep 14 02:53:29.889168 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gqv2\" (UniqueName: \"kubernetes.io/projected/d642bccd-dc72-4526-8304-fa4f74691d91-kube-api-access-5gqv2\") pod \"downloads-76f99584f4-p7bpw\" (UID: \"d642bccd-dc72-4526-8304-fa4f74691d91\") " pod="openshift-console/downloads-76f99584f4-p7bpw"
Sep 14 02:53:29.889261 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a09d4a75-a151-43e2-bab0-50eb005b2776-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69957b6f78-nxzq8\" (UID: \"a09d4a75-a151-43e2-bab0-50eb005b2776\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8"
Sep 14 02:53:29.889261 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-serving-cert\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.889261 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scwp2\" (UniqueName: \"kubernetes.io/projected/a09d4a75-a151-43e2-bab0-50eb005b2776-kube-api-access-scwp2\") pod \"managed-serviceaccount-addon-agent-69957b6f78-nxzq8\" (UID: \"a09d4a75-a151-43e2-bab0-50eb005b2776\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8"
Sep 14 02:53:29.889410 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2sgz\" (UniqueName: \"kubernetes.io/projected/6e1355a1-2166-47d0-a829-923ef50e7c7a-kube-api-access-v2sgz\") pod \"klusterlet-addon-workmgr-586c8749df-vtb4g\" (UID: \"6e1355a1-2166-47d0-a829-923ef50e7c7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:29.889410 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-service-ca\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.889410 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e1355a1-2166-47d0-a829-923ef50e7c7a-tmp\") pod \"klusterlet-addon-workmgr-586c8749df-vtb4g\" (UID: \"6e1355a1-2166-47d0-a829-923ef50e7c7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:29.889410 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6e1355a1-2166-47d0-a829-923ef50e7c7a-klusterlet-config\") pod \"klusterlet-addon-workmgr-586c8749df-vtb4g\" (UID: \"6e1355a1-2166-47d0-a829-923ef50e7c7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:29.889410 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-oauth-config\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.889410 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqztk\" (UniqueName: \"kubernetes.io/projected/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-kube-api-access-lqztk\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.889642 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-config\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.889642 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-oauth-serving-cert\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.889704 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.889664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e1355a1-2166-47d0-a829-923ef50e7c7a-tmp\") pod \"klusterlet-addon-workmgr-586c8749df-vtb4g\" (UID: \"6e1355a1-2166-47d0-a829-923ef50e7c7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:29.891663 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.891645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6e1355a1-2166-47d0-a829-923ef50e7c7a-klusterlet-config\") pod \"klusterlet-addon-workmgr-586c8749df-vtb4g\" (UID: \"6e1355a1-2166-47d0-a829-923ef50e7c7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:29.897978 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.897960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2sgz\" (UniqueName: \"kubernetes.io/projected/6e1355a1-2166-47d0-a829-923ef50e7c7a-kube-api-access-v2sgz\") pod \"klusterlet-addon-workmgr-586c8749df-vtb4g\" (UID: \"6e1355a1-2166-47d0-a829-923ef50e7c7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:29.990095 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.990017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-service-ca\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.990095 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.990087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-oauth-config\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.990238 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.990104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqztk\" (UniqueName: \"kubernetes.io/projected/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-kube-api-access-lqztk\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.990238 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.990124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-config\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.990238 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.990146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-oauth-serving-cert\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.990238 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.990171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gqv2\" (UniqueName: \"kubernetes.io/projected/d642bccd-dc72-4526-8304-fa4f74691d91-kube-api-access-5gqv2\") pod \"downloads-76f99584f4-p7bpw\" (UID: \"d642bccd-dc72-4526-8304-fa4f74691d91\") " pod="openshift-console/downloads-76f99584f4-p7bpw"
Sep 14 02:53:29.990238 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.990201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a09d4a75-a151-43e2-bab0-50eb005b2776-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69957b6f78-nxzq8\" (UID: \"a09d4a75-a151-43e2-bab0-50eb005b2776\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8"
Sep 14 02:53:29.990483 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.990262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-serving-cert\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.990483 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.990308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scwp2\" (UniqueName: \"kubernetes.io/projected/a09d4a75-a151-43e2-bab0-50eb005b2776-kube-api-access-scwp2\") pod \"managed-serviceaccount-addon-agent-69957b6f78-nxzq8\" (UID: \"a09d4a75-a151-43e2-bab0-50eb005b2776\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8"
Sep 14 02:53:29.990682 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.990656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-service-ca\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.991174 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.991150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-config\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.991174 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.991171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-oauth-serving-cert\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.992615 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.992586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-serving-cert\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:29.992716 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.992595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a09d4a75-a151-43e2-bab0-50eb005b2776-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69957b6f78-nxzq8\" (UID: \"a09d4a75-a151-43e2-bab0-50eb005b2776\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8"
Sep 14 02:53:29.992716 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:29.992632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-oauth-config\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:30.002518 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.002498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gqv2\" (UniqueName: \"kubernetes.io/projected/d642bccd-dc72-4526-8304-fa4f74691d91-kube-api-access-5gqv2\") pod \"downloads-76f99584f4-p7bpw\" (UID: \"d642bccd-dc72-4526-8304-fa4f74691d91\") " pod="openshift-console/downloads-76f99584f4-p7bpw"
Sep 14 02:53:30.003855 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.003834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqztk\" (UniqueName: \"kubernetes.io/projected/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-kube-api-access-lqztk\") pod \"console-565bccd9d-thp2r\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") " pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:30.003937 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.003871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scwp2\" (UniqueName: \"kubernetes.io/projected/a09d4a75-a151-43e2-bab0-50eb005b2776-kube-api-access-scwp2\") pod \"managed-serviceaccount-addon-agent-69957b6f78-nxzq8\" (UID: \"a09d4a75-a151-43e2-bab0-50eb005b2776\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8"
Sep 14 02:53:30.039935 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.039913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:30.072048 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.072002    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8"
Sep 14 02:53:30.109296 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.109180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:30.112814 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.112798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-p7bpw"
Sep 14 02:53:30.236140 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.236103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgmx4" event={"ID":"fb5579be-c3dc-43f1-aa7c-3bd709206921","Type":"ContainerStarted","Data":"5843a85f1e24f1b8f67dbf2cd9b27b7c9d3add955da17b9d281e2cfeee34d96f"}
Sep 14 02:53:30.258934 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.258892    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hgmx4" podStartSLOduration=34.163515671 podStartE2EDuration="36.258876162s" totalImagesPullingTime="2.095360491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:30.257635077 +0000 UTC m=+76.983593144" watchObservedRunningTime="2026-09-14 02:53:30.258876162 +0000 UTC m=+76.984834214"
Sep 14 02:53:30.307092 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.307071    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"]
Sep 14 02:53:30.308888 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:30.308867    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1355a1_2166_47d0_a829_923ef50e7c7a.slice/crio-29d3105d9d4b69d811d8188dd34f848e1ecec3027eb6421a16eaa64b4856b9f2 WatchSource:0}: Error finding container 29d3105d9d4b69d811d8188dd34f848e1ecec3027eb6421a16eaa64b4856b9f2: Status 404 returned error can't find the container with id 29d3105d9d4b69d811d8188dd34f848e1ecec3027eb6421a16eaa64b4856b9f2
Sep 14 02:53:30.347169 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.347146    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8"]
Sep 14 02:53:30.350184 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:30.350154    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda09d4a75_a151_43e2_bab0_50eb005b2776.slice/crio-f1ef5f0d7f636529fe4edf126a56104cf6c50b5c59b76b3f5e0a8a2158f67043 WatchSource:0}: Error finding container f1ef5f0d7f636529fe4edf126a56104cf6c50b5c59b76b3f5e0a8a2158f67043: Status 404 returned error can't find the container with id f1ef5f0d7f636529fe4edf126a56104cf6c50b5c59b76b3f5e0a8a2158f67043
Sep 14 02:53:30.395193 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.395169    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-p7bpw"]
Sep 14 02:53:30.396345 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:30.396327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565bccd9d-thp2r"]
Sep 14 02:53:30.397539 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:30.397517    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd642bccd_dc72_4526_8304_fa4f74691d91.slice/crio-a71f07d8c7dcdaa02abbab07e694391b4e320d7b97b28af40b5a11fa780559da WatchSource:0}: Error finding container a71f07d8c7dcdaa02abbab07e694391b4e320d7b97b28af40b5a11fa780559da: Status 404 returned error can't find the container with id a71f07d8c7dcdaa02abbab07e694391b4e320d7b97b28af40b5a11fa780559da
Sep 14 02:53:30.400242 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:30.400220    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26a890a_c22f_45ef_a3e0_ae89bd05aa87.slice/crio-cacbdaae1be731281cd872d6ebe070edd4ee1d5b8fd22a086049367c90bc69ec WatchSource:0}: Error finding container cacbdaae1be731281cd872d6ebe070edd4ee1d5b8fd22a086049367c90bc69ec: Status 404 returned error can't find the container with id cacbdaae1be731281cd872d6ebe070edd4ee1d5b8fd22a086049367c90bc69ec
Sep 14 02:53:31.242794 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:31.242758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-p7bpw" event={"ID":"d642bccd-dc72-4526-8304-fa4f74691d91","Type":"ContainerStarted","Data":"a71f07d8c7dcdaa02abbab07e694391b4e320d7b97b28af40b5a11fa780559da"}
Sep 14 02:53:31.244517 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:31.244429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565bccd9d-thp2r" event={"ID":"c26a890a-c22f-45ef-a3e0-ae89bd05aa87","Type":"ContainerStarted","Data":"cacbdaae1be731281cd872d6ebe070edd4ee1d5b8fd22a086049367c90bc69ec"}
Sep 14 02:53:31.246132 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:31.246069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8" event={"ID":"a09d4a75-a151-43e2-bab0-50eb005b2776","Type":"ContainerStarted","Data":"f1ef5f0d7f636529fe4edf126a56104cf6c50b5c59b76b3f5e0a8a2158f67043"}
Sep 14 02:53:31.252138 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:31.252084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g" event={"ID":"6e1355a1-2166-47d0-a829-923ef50e7c7a","Type":"ContainerStarted","Data":"29d3105d9d4b69d811d8188dd34f848e1ecec3027eb6421a16eaa64b4856b9f2"}
Sep 14 02:53:37.229135 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:37.229109    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lqmmv"
Sep 14 02:53:38.291380 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:38.291219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g" event={"ID":"6e1355a1-2166-47d0-a829-923ef50e7c7a","Type":"ContainerStarted","Data":"3fde8a7e6ec17aeaa194d8a2f9b537a276d1aed081547e0aa9ed1a6ed0836f8e"}
Sep 14 02:53:38.291380 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:38.291344    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:38.293329 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:38.293300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565bccd9d-thp2r" event={"ID":"c26a890a-c22f-45ef-a3e0-ae89bd05aa87","Type":"ContainerStarted","Data":"1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8"}
Sep 14 02:53:38.293488 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:38.293463    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g"
Sep 14 02:53:38.295131 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:38.295104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8" event={"ID":"a09d4a75-a151-43e2-bab0-50eb005b2776","Type":"ContainerStarted","Data":"66a07fb3f70d2266570a1d21e260e9f542d3e4c3ff23e3898753f8277cd74697"}
Sep 14 02:53:38.309372 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:38.309313    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-586c8749df-vtb4g" podStartSLOduration=2.393037636 podStartE2EDuration="9.309296133s" totalImagesPullingTime="6.916258497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:38.307099844 +0000 UTC m=+85.033057898" watchObservedRunningTime="2026-09-14 02:53:38.309296133 +0000 UTC m=+85.035254187"
Sep 14 02:53:38.339661 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:38.339613    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69957b6f78-nxzq8" podStartSLOduration=2.492200199 podStartE2EDuration="9.3396004s" totalImagesPullingTime="6.847400201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:38.338109668 +0000 UTC m=+85.064067732" watchObservedRunningTime="2026-09-14 02:53:38.3396004 +0000 UTC m=+85.065558456"
Sep 14 02:53:38.357009 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:38.356960    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-565bccd9d-thp2r" podStartSLOduration=2.550397421 podStartE2EDuration="9.356950686s" totalImagesPullingTime="6.806553265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:38.356713154 +0000 UTC m=+85.082671206" watchObservedRunningTime="2026-09-14 02:53:38.356950686 +0000 UTC m=+85.082908738"
Sep 14 02:53:39.785810 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:39.785776    2580 patch_prober.go:28] interesting pod/image-registry-6fcc6d8f6b-qz6n5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 14 02:53:39.786253 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:39.785834    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5" podUID="2c457bc4-98d8-4902-9960-6976161569e1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 02:53:40.110062 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:40.109965    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:40.110062 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:40.110051    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:40.115476 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:40.115448    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:40.309470 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:40.309432    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:53:41.512005 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.511969    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xmkc6"]
Sep 14 02:53:41.516590 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.516566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.519857 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.519837    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 14 02:53:41.523504 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.523329    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xb6b2\""
Sep 14 02:53:41.523702 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.523668    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 14 02:53:41.523702 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.523668    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 14 02:53:41.523900 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.523776    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 14 02:53:41.694952 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.694915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.695141 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.694971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-tls\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.695141 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.695064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-sys\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.695141 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.695081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22hcl\" (UniqueName: \"kubernetes.io/projected/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-kube-api-access-22hcl\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.695141 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.695098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-wtmp\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.695141 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.695115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-metrics-client-ca\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.695141 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.695137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-root\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.695383 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.695157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-textfile\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.695383 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.695198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796263 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-tls\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796263 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-sys\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796495 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22hcl\" (UniqueName: \"kubernetes.io/projected/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-kube-api-access-22hcl\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796495 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-wtmp\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796495 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-metrics-client-ca\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796495 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-root\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796495 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-sys\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796495 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-textfile\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796495 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796797 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796797 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-textfile\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796797 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-wtmp\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.796797 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.796759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-root\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.797188 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.797112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-metrics-client-ca\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.797310 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.797214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.799110 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.799089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-tls\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.799213 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.799180    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.809481 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.809459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22hcl\" (UniqueName: \"kubernetes.io/projected/dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b-kube-api-access-22hcl\") pod \"node-exporter-xmkc6\" (UID: \"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b\") " pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:41.826990 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:41.826966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xmkc6"
Sep 14 02:53:43.203040 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:43.203006    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:53:45.483916 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:45.483883    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-565bccd9d-thp2r"]
Sep 14 02:53:48.018025 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:53:48.017990    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd2c10a_4113_42c7_88b2_3dfa8bad9d7b.slice/crio-6766a5fb1085a4725e00a9f5e0382ec2a6daf9e6ecc8b2ed534d0c645caa672e WatchSource:0}: Error finding container 6766a5fb1085a4725e00a9f5e0382ec2a6daf9e6ecc8b2ed534d0c645caa672e: Status 404 returned error can't find the container with id 6766a5fb1085a4725e00a9f5e0382ec2a6daf9e6ecc8b2ed534d0c645caa672e
Sep 14 02:53:48.336901 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:48.336827    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmkc6" event={"ID":"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b","Type":"ContainerStarted","Data":"6766a5fb1085a4725e00a9f5e0382ec2a6daf9e6ecc8b2ed534d0c645caa672e"}
Sep 14 02:53:49.343767 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:49.343729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-p7bpw" event={"ID":"d642bccd-dc72-4526-8304-fa4f74691d91","Type":"ContainerStarted","Data":"763df40fe1390f14d5223adec9b5d6ec837d325ecfbc89d1174cc5d7c9e809fe"}
Sep 14 02:53:49.344285 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:49.343950    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-p7bpw"
Sep 14 02:53:49.361936 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:49.361881    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-p7bpw" podStartSLOduration=2.375271469 podStartE2EDuration="20.361863178s" totalImagesPullingTime="17.986591709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:49.360690849 +0000 UTC m=+96.086648901" watchObservedRunningTime="2026-09-14 02:53:49.361863178 +0000 UTC m=+96.087821231"
Sep 14 02:53:49.364913 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:49.364889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-p7bpw"
Sep 14 02:53:50.350356 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:50.350317    2580 generic.go:356] "Generic (PLEG): container finished" podID="dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b" containerID="167350bf0083f4738eca8fd2c7884a1a826ccf130e7c2f7826399d9268afd12b" exitCode=0
Sep 14 02:53:50.350807 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:50.350419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmkc6" event={"ID":"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b","Type":"ContainerDied","Data":"167350bf0083f4738eca8fd2c7884a1a826ccf130e7c2f7826399d9268afd12b"}
Sep 14 02:53:51.357007 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:51.356965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmkc6" event={"ID":"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b","Type":"ContainerStarted","Data":"98d9aced509156bc693f7f432d29b68006ab123c88b5102d27ba0ab81c523cf8"}
Sep 14 02:53:51.357498 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:51.357055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmkc6" event={"ID":"dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b","Type":"ContainerStarted","Data":"11f96a6b5e2322bf4baa9cf177ad2a610602f635383b44799d3c7d0f2413f938"}
Sep 14 02:53:51.376399 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:51.376352    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xmkc6" podStartSLOduration=8.793206524 podStartE2EDuration="10.376333794s" totalImagesPullingTime="1.502156707s" totalInitContainerRuntime="80.970563ms" isStatefulPod=true podCreationTimestamp="2026-09-14 02:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:51.375339562 +0000 UTC m=+98.101297627" watchObservedRunningTime="2026-09-14 02:53:51.376333794 +0000 UTC m=+98.102291846"
Sep 14 02:53:52.251569 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:52.251535    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"]
Sep 14 02:53:58.229559 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:53:58.229526    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g4jlk"
Sep 14 02:54:10.504126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.504092    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-565bccd9d-thp2r" podUID="c26a890a-c22f-45ef-a3e0-ae89bd05aa87" containerName="console" containerID="cri-o://1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8" gracePeriod=15
Sep 14 02:54:10.771649 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.771626    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:54:10.808898 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.808870    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqztk\" (UniqueName: \"kubernetes.io/projected/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-kube-api-access-lqztk\") pod \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") "
Sep 14 02:54:10.809019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.808912    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-oauth-serving-cert\") pod \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") "
Sep 14 02:54:10.809019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.808958    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-config\") pod \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") "
Sep 14 02:54:10.809019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.808993    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-serving-cert\") pod \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") "
Sep 14 02:54:10.809019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.809015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-service-ca\") pod \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") "
Sep 14 02:54:10.809250 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.809071    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-oauth-config\") pod \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\" (UID: \"c26a890a-c22f-45ef-a3e0-ae89bd05aa87\") "
Sep 14 02:54:10.809425 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.809371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c26a890a-c22f-45ef-a3e0-ae89bd05aa87" (UID: "c26a890a-c22f-45ef-a3e0-ae89bd05aa87"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:54:10.809425 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.809396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-config" (OuterVolumeSpecName: "console-config") pod "c26a890a-c22f-45ef-a3e0-ae89bd05aa87" (UID: "c26a890a-c22f-45ef-a3e0-ae89bd05aa87"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:54:10.809584 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.809445    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-service-ca" (OuterVolumeSpecName: "service-ca") pod "c26a890a-c22f-45ef-a3e0-ae89bd05aa87" (UID: "c26a890a-c22f-45ef-a3e0-ae89bd05aa87"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:54:10.811214 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.811187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c26a890a-c22f-45ef-a3e0-ae89bd05aa87" (UID: "c26a890a-c22f-45ef-a3e0-ae89bd05aa87"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:54:10.811214 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.811202    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-kube-api-access-lqztk" (OuterVolumeSpecName: "kube-api-access-lqztk") pod "c26a890a-c22f-45ef-a3e0-ae89bd05aa87" (UID: "c26a890a-c22f-45ef-a3e0-ae89bd05aa87"). InnerVolumeSpecName "kube-api-access-lqztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:54:10.811613 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.811590    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c26a890a-c22f-45ef-a3e0-ae89bd05aa87" (UID: "c26a890a-c22f-45ef-a3e0-ae89bd05aa87"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:54:10.909776 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.909747    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-config\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:10.909776 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.909771    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-serving-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:10.909776 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.909781    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:10.909956 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.909790    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-console-oauth-config\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:10.909956 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.909799    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqztk\" (UniqueName: \"kubernetes.io/projected/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-kube-api-access-lqztk\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:10.909956 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:10.909807    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26a890a-c22f-45ef-a3e0-ae89bd05aa87-oauth-serving-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:11.430483 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:11.430449    2580 generic.go:356] "Generic (PLEG): container finished" podID="c26a890a-c22f-45ef-a3e0-ae89bd05aa87" containerID="1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8" exitCode=0
Sep 14 02:54:11.430671 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:11.430488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565bccd9d-thp2r" event={"ID":"c26a890a-c22f-45ef-a3e0-ae89bd05aa87","Type":"ContainerDied","Data":"1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8"}
Sep 14 02:54:11.430671 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:11.430532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565bccd9d-thp2r" event={"ID":"c26a890a-c22f-45ef-a3e0-ae89bd05aa87","Type":"ContainerDied","Data":"cacbdaae1be731281cd872d6ebe070edd4ee1d5b8fd22a086049367c90bc69ec"}
Sep 14 02:54:11.430671 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:11.430559    2580 scope.go:117] "RemoveContainer" containerID="1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8"
Sep 14 02:54:11.430671 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:11.430492    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565bccd9d-thp2r"
Sep 14 02:54:11.438995 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:11.438978    2580 scope.go:117] "RemoveContainer" containerID="1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8"
Sep 14 02:54:11.439281 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:54:11.439260    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8\": container with ID starting with 1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8 not found: ID does not exist" containerID="1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8"
Sep 14 02:54:11.439352 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:11.439292    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8"} err="failed to get container status \"1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8\": rpc error: code = NotFound desc = could not find container \"1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8\": container with ID starting with 1e4eb46c387a122fef115319eca8cf50fc1661b500c88f12ad0faf648e3e0ea8 not found: ID does not exist"
Sep 14 02:54:11.461194 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:11.460169    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-565bccd9d-thp2r"]
Sep 14 02:54:11.463177 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:11.463155    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-565bccd9d-thp2r"]
Sep 14 02:54:11.755316 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:11.755289    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c26a890a-c22f-45ef-a3e0-ae89bd05aa87" path="/var/lib/kubelet/pods/c26a890a-c22f-45ef-a3e0-ae89bd05aa87/volumes"
Sep 14 02:54:14.446024 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:14.445942    2580 generic.go:356] "Generic (PLEG): container finished" podID="1a167d62-c256-4cc5-a3e6-5f6eb19cdb15" containerID="c5ddecd3498c52b886391ede1ed57d2adcacb7217f31a6cde0db43c25af294e4" exitCode=0
Sep 14 02:54:14.446456 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:14.446013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kmjvv" event={"ID":"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15","Type":"ContainerDied","Data":"c5ddecd3498c52b886391ede1ed57d2adcacb7217f31a6cde0db43c25af294e4"}
Sep 14 02:54:14.446521 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:14.446483    2580 scope.go:117] "RemoveContainer" containerID="c5ddecd3498c52b886391ede1ed57d2adcacb7217f31a6cde0db43c25af294e4"
Sep 14 02:54:15.456468 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:15.455192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kmjvv" event={"ID":"1a167d62-c256-4cc5-a3e6-5f6eb19cdb15","Type":"ContainerStarted","Data":"4c6b5992fbeeb74277a108b8cb08814e9482001d619a0866d3925993eac5bf46"}
Sep 14 02:54:17.272989 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.272935    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5" podUID="2c457bc4-98d8-4902-9960-6976161569e1" containerName="registry" containerID="cri-o://32fa618c4c731a5f38b5c5c3335fd21e0f12751525e5c417ac6df328b186c63b" gracePeriod=30
Sep 14 02:54:17.464847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.464590    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c457bc4-98d8-4902-9960-6976161569e1" containerID="32fa618c4c731a5f38b5c5c3335fd21e0f12751525e5c417ac6df328b186c63b" exitCode=0
Sep 14 02:54:17.464847 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.464698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5" event={"ID":"2c457bc4-98d8-4902-9960-6976161569e1","Type":"ContainerDied","Data":"32fa618c4c731a5f38b5c5c3335fd21e0f12751525e5c417ac6df328b186c63b"}
Sep 14 02:54:17.546197 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.546147    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:54:17.655058 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.655018    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls\") pod \"2c457bc4-98d8-4902-9960-6976161569e1\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") "
Sep 14 02:54:17.655183 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.655069    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh4jx\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-kube-api-access-hh4jx\") pod \"2c457bc4-98d8-4902-9960-6976161569e1\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") "
Sep 14 02:54:17.655183 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.655091    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-bound-sa-token\") pod \"2c457bc4-98d8-4902-9960-6976161569e1\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") "
Sep 14 02:54:17.655183 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.655113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-image-registry-private-configuration\") pod \"2c457bc4-98d8-4902-9960-6976161569e1\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") "
Sep 14 02:54:17.655183 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.655137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-trusted-ca\") pod \"2c457bc4-98d8-4902-9960-6976161569e1\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") "
Sep 14 02:54:17.655183 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.655155    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-registry-certificates\") pod \"2c457bc4-98d8-4902-9960-6976161569e1\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") "
Sep 14 02:54:17.655183 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.655174    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-installation-pull-secrets\") pod \"2c457bc4-98d8-4902-9960-6976161569e1\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") "
Sep 14 02:54:17.655496 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.655191    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c457bc4-98d8-4902-9960-6976161569e1-ca-trust-extracted\") pod \"2c457bc4-98d8-4902-9960-6976161569e1\" (UID: \"2c457bc4-98d8-4902-9960-6976161569e1\") "
Sep 14 02:54:17.655692 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.655643    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2c457bc4-98d8-4902-9960-6976161569e1" (UID: "2c457bc4-98d8-4902-9960-6976161569e1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:54:17.655811 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.655715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2c457bc4-98d8-4902-9960-6976161569e1" (UID: "2c457bc4-98d8-4902-9960-6976161569e1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:54:17.658132 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.658087    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2c457bc4-98d8-4902-9960-6976161569e1" (UID: "2c457bc4-98d8-4902-9960-6976161569e1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:54:17.658132 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.658095    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2c457bc4-98d8-4902-9960-6976161569e1" (UID: "2c457bc4-98d8-4902-9960-6976161569e1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:54:17.658270 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.658152    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-kube-api-access-hh4jx" (OuterVolumeSpecName: "kube-api-access-hh4jx") pod "2c457bc4-98d8-4902-9960-6976161569e1" (UID: "2c457bc4-98d8-4902-9960-6976161569e1"). InnerVolumeSpecName "kube-api-access-hh4jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:54:17.658270 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.658159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2c457bc4-98d8-4902-9960-6976161569e1" (UID: "2c457bc4-98d8-4902-9960-6976161569e1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:54:17.658270 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.658251    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2c457bc4-98d8-4902-9960-6976161569e1" (UID: "2c457bc4-98d8-4902-9960-6976161569e1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:54:17.663921 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.663901    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c457bc4-98d8-4902-9960-6976161569e1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2c457bc4-98d8-4902-9960-6976161569e1" (UID: "2c457bc4-98d8-4902-9960-6976161569e1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:54:17.756137 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.756114    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-registry-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:17.756137 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.756136    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh4jx\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-kube-api-access-hh4jx\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:17.756278 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.756149    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c457bc4-98d8-4902-9960-6976161569e1-bound-sa-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:17.756278 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.756162    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-image-registry-private-configuration\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:17.756278 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.756175    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-trusted-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:17.756278 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.756186    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c457bc4-98d8-4902-9960-6976161569e1-registry-certificates\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:17.756278 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.756200    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c457bc4-98d8-4902-9960-6976161569e1-installation-pull-secrets\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:17.756278 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:17.756213    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c457bc4-98d8-4902-9960-6976161569e1-ca-trust-extracted\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:54:18.469159 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:18.469132    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"
Sep 14 02:54:18.469550 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:18.469134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5" event={"ID":"2c457bc4-98d8-4902-9960-6976161569e1","Type":"ContainerDied","Data":"696ff03025278d20be8b2ef65376e75043df56c3199a26edacd59b67b30bc992"}
Sep 14 02:54:18.469550 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:18.469266    2580 scope.go:117] "RemoveContainer" containerID="32fa618c4c731a5f38b5c5c3335fd21e0f12751525e5c417ac6df328b186c63b"
Sep 14 02:54:18.487152 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:18.487126    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"]
Sep 14 02:54:18.490914 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:18.490891    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fcc6d8f6b-qz6n5"]
Sep 14 02:54:19.754893 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:54:19.754858    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c457bc4-98d8-4902-9960-6976161569e1" path="/var/lib/kubelet/pods/2c457bc4-98d8-4902-9960-6976161569e1/volumes"
Sep 14 02:55:09.061715 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.061631    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-656d967696-phm4z"]
Sep 14 02:55:09.062212 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.061868    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26a890a-c22f-45ef-a3e0-ae89bd05aa87" containerName="console"
Sep 14 02:55:09.062212 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.061879    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26a890a-c22f-45ef-a3e0-ae89bd05aa87" containerName="console"
Sep 14 02:55:09.062212 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.061894    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c457bc4-98d8-4902-9960-6976161569e1" containerName="registry"
Sep 14 02:55:09.062212 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.061899    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c457bc4-98d8-4902-9960-6976161569e1" containerName="registry"
Sep 14 02:55:09.062212 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.061942    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26a890a-c22f-45ef-a3e0-ae89bd05aa87" containerName="console"
Sep 14 02:55:09.062212 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.061949    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c457bc4-98d8-4902-9960-6976161569e1" containerName="registry"
Sep 14 02:55:09.064040 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.063996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.067409 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.067391    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xzm62\""
Sep 14 02:55:09.067762 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.067742    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 14 02:55:09.068141 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.068119    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 14 02:55:09.068224 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.068128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 14 02:55:09.068224 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.068176    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 14 02:55:09.069801 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.069780    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 14 02:55:09.092440 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.092421    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-656d967696-phm4z"]
Sep 14 02:55:09.095504 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.095482    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 14 02:55:09.213450 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.213421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-oauth-serving-cert\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.213559 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.213452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8k7m\" (UniqueName: \"kubernetes.io/projected/277fa9ad-963b-4899-8628-1ef9371ec9a2-kube-api-access-m8k7m\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.213559 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.213475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-service-ca\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.213559 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.213548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-oauth-config\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.213663 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.213581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-config\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.213663 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.213608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-serving-cert\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.213663 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.213624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-trusted-ca-bundle\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.314297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.314243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-service-ca\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.314297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.314277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-oauth-config\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.314297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.314294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-config\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.314508 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.314320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-serving-cert\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.314508 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.314338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-trusted-ca-bundle\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.314508 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.314370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-oauth-serving-cert\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.314508 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.314398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8k7m\" (UniqueName: \"kubernetes.io/projected/277fa9ad-963b-4899-8628-1ef9371ec9a2-kube-api-access-m8k7m\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.315054 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.315010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-service-ca\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.315153 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.315100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-oauth-serving-cert\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.315153 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.315105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-config\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.315267 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.315234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-trusted-ca-bundle\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.316841 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.316817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-oauth-config\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.316919 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.316861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-serving-cert\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.323825 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.323797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8k7m\" (UniqueName: \"kubernetes.io/projected/277fa9ad-963b-4899-8628-1ef9371ec9a2-kube-api-access-m8k7m\") pod \"console-656d967696-phm4z\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") " pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.372786 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.372758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:09.646674 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:09.646595    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-656d967696-phm4z"]
Sep 14 02:55:09.649630 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:55:09.649602    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277fa9ad_963b_4899_8628_1ef9371ec9a2.slice/crio-45a3e63c9962d321c5773f3989e963e1ad1754f8288bfc4a80e84975e543f142 WatchSource:0}: Error finding container 45a3e63c9962d321c5773f3989e963e1ad1754f8288bfc4a80e84975e543f142: Status 404 returned error can't find the container with id 45a3e63c9962d321c5773f3989e963e1ad1754f8288bfc4a80e84975e543f142
Sep 14 02:55:10.653485 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:10.653450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656d967696-phm4z" event={"ID":"277fa9ad-963b-4899-8628-1ef9371ec9a2","Type":"ContainerStarted","Data":"3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5"}
Sep 14 02:55:10.653858 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:10.653527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656d967696-phm4z" event={"ID":"277fa9ad-963b-4899-8628-1ef9371ec9a2","Type":"ContainerStarted","Data":"45a3e63c9962d321c5773f3989e963e1ad1754f8288bfc4a80e84975e543f142"}
Sep 14 02:55:10.679315 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:10.679269    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-656d967696-phm4z" podStartSLOduration=1.679254641 podStartE2EDuration="1.679254641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:55:10.678556895 +0000 UTC m=+177.404514948" watchObservedRunningTime="2026-09-14 02:55:10.679254641 +0000 UTC m=+177.405212753"
Sep 14 02:55:19.372865 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:19.372830    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:19.373247 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:19.372888    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:19.377533 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:19.377510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:55:19.693210 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:55:19.693137    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:56:20.021205 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.021139    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c576996f7-j7d57"]
Sep 14 02:56:20.023705 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.023680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.040914 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.040889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c576996f7-j7d57"]
Sep 14 02:56:20.086698 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.086672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-oauth-config\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.086826 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.086705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-oauth-serving-cert\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.086826 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.086737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-service-ca\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.086826 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.086775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-trusted-ca-bundle\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.086826 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.086806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-console-config\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.086959 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.086853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh8q4\" (UniqueName: \"kubernetes.io/projected/dccb7399-dba3-4e76-b861-5a109fff882c-kube-api-access-xh8q4\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.086959 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.086893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-serving-cert\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.187360 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.187331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-oauth-config\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.187519 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.187370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-oauth-serving-cert\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.187519 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.187394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-service-ca\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.187519 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.187414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-trusted-ca-bundle\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.187519 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.187448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-console-config\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.187519 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.187472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh8q4\" (UniqueName: \"kubernetes.io/projected/dccb7399-dba3-4e76-b861-5a109fff882c-kube-api-access-xh8q4\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.187519 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.187499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-serving-cert\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.188219 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.188187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-service-ca\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.188219 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.188207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-oauth-serving-cert\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.188364 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.188231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-console-config\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.188399 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.188369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-trusted-ca-bundle\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.189839 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.189815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-serving-cert\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.189917 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.189867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-oauth-config\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.196769 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.196748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh8q4\" (UniqueName: \"kubernetes.io/projected/dccb7399-dba3-4e76-b861-5a109fff882c-kube-api-access-xh8q4\") pod \"console-7c576996f7-j7d57\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") " pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.333607 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.333548    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:20.607417 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.607323    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c576996f7-j7d57"]
Sep 14 02:56:20.609864 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:56:20.609831    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccb7399_dba3_4e76_b861_5a109fff882c.slice/crio-cd04cdce6baf205cfdfce0cb34b2aa9b31fcbb28732d0e45c0ea3d7420e1b262 WatchSource:0}: Error finding container cd04cdce6baf205cfdfce0cb34b2aa9b31fcbb28732d0e45c0ea3d7420e1b262: Status 404 returned error can't find the container with id cd04cdce6baf205cfdfce0cb34b2aa9b31fcbb28732d0e45c0ea3d7420e1b262
Sep 14 02:56:20.897161 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.897079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c576996f7-j7d57" event={"ID":"dccb7399-dba3-4e76-b861-5a109fff882c","Type":"ContainerStarted","Data":"fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636"}
Sep 14 02:56:20.897161 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.897154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c576996f7-j7d57" event={"ID":"dccb7399-dba3-4e76-b861-5a109fff882c","Type":"ContainerStarted","Data":"cd04cdce6baf205cfdfce0cb34b2aa9b31fcbb28732d0e45c0ea3d7420e1b262"}
Sep 14 02:56:20.944774 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:20.944727    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c576996f7-j7d57" podStartSLOduration=1.9447127119999998 podStartE2EDuration="1.944712712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:56:20.942945477 +0000 UTC m=+247.668903530" watchObservedRunningTime="2026-09-14 02:56:20.944712712 +0000 UTC m=+247.670670764"
Sep 14 02:56:30.333879 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:30.333847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:30.334282 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:30.333927    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:30.338676 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:30.338654    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:30.936335 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:30.936308    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:56:30.991192 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:30.991161    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-656d967696-phm4z"]
Sep 14 02:56:54.585608 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.585575    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"]
Sep 14 02:56:54.587803 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.587787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:56:54.591129 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.591110    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c2rkr\""
Sep 14 02:56:54.591494 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.591480    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 02:56:54.591727 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.591711    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 02:56:54.600560 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.600538    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"]
Sep 14 02:56:54.729516 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.729490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbdlf\" (UniqueName: \"kubernetes.io/projected/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-kube-api-access-fbdlf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:56:54.729635 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.729523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:56:54.729635 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.729555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:56:54.829838 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.829814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbdlf\" (UniqueName: \"kubernetes.io/projected/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-kube-api-access-fbdlf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:56:54.829926 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.829847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:56:54.829926 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.829889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:56:54.830290 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.830270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:56:54.830374 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.830331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:56:54.838366 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.838326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbdlf\" (UniqueName: \"kubernetes.io/projected/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-kube-api-access-fbdlf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:56:54.897123 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:54.897100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:56:55.162903 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:55.162832    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"]
Sep 14 02:56:55.166665 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:56:55.166635    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dc5e8a_6ece_4805_b005_0f8e70ff1955.slice/crio-8f3669a9727fb7727ba9050b8c82751e8cb11d837c0b68c9dd671bab23345f38 WatchSource:0}: Error finding container 8f3669a9727fb7727ba9050b8c82751e8cb11d837c0b68c9dd671bab23345f38: Status 404 returned error can't find the container with id 8f3669a9727fb7727ba9050b8c82751e8cb11d837c0b68c9dd671bab23345f38
Sep 14 02:56:56.011496 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.011425    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-656d967696-phm4z" podUID="277fa9ad-963b-4899-8628-1ef9371ec9a2" containerName="console" containerID="cri-o://3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5" gracePeriod=15
Sep 14 02:56:56.024929 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.024903    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s" event={"ID":"b3dc5e8a-6ece-4805-b005-0f8e70ff1955","Type":"ContainerStarted","Data":"8f3669a9727fb7727ba9050b8c82751e8cb11d837c0b68c9dd671bab23345f38"}
Sep 14 02:56:56.261976 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.261921    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:56:56.343700 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.343673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-serving-cert\") pod \"277fa9ad-963b-4899-8628-1ef9371ec9a2\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") "
Sep 14 02:56:56.343875 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.343711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-trusted-ca-bundle\") pod \"277fa9ad-963b-4899-8628-1ef9371ec9a2\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") "
Sep 14 02:56:56.343875 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.343747    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-oauth-serving-cert\") pod \"277fa9ad-963b-4899-8628-1ef9371ec9a2\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") "
Sep 14 02:56:56.343990 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.343867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-config\") pod \"277fa9ad-963b-4899-8628-1ef9371ec9a2\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") "
Sep 14 02:56:56.343990 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.343934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-oauth-config\") pod \"277fa9ad-963b-4899-8628-1ef9371ec9a2\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") "
Sep 14 02:56:56.343990 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.343974    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-service-ca\") pod \"277fa9ad-963b-4899-8628-1ef9371ec9a2\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") "
Sep 14 02:56:56.344158 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.344047    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8k7m\" (UniqueName: \"kubernetes.io/projected/277fa9ad-963b-4899-8628-1ef9371ec9a2-kube-api-access-m8k7m\") pod \"277fa9ad-963b-4899-8628-1ef9371ec9a2\" (UID: \"277fa9ad-963b-4899-8628-1ef9371ec9a2\") "
Sep 14 02:56:56.344300 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.344188    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "277fa9ad-963b-4899-8628-1ef9371ec9a2" (UID: "277fa9ad-963b-4899-8628-1ef9371ec9a2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:56:56.344300 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.344177    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "277fa9ad-963b-4899-8628-1ef9371ec9a2" (UID: "277fa9ad-963b-4899-8628-1ef9371ec9a2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:56:56.344487 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.344299    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-config" (OuterVolumeSpecName: "console-config") pod "277fa9ad-963b-4899-8628-1ef9371ec9a2" (UID: "277fa9ad-963b-4899-8628-1ef9371ec9a2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:56:56.344487 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.344371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-service-ca" (OuterVolumeSpecName: "service-ca") pod "277fa9ad-963b-4899-8628-1ef9371ec9a2" (UID: "277fa9ad-963b-4899-8628-1ef9371ec9a2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:56:56.345991 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.345962    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "277fa9ad-963b-4899-8628-1ef9371ec9a2" (UID: "277fa9ad-963b-4899-8628-1ef9371ec9a2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:56:56.346104 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.346011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "277fa9ad-963b-4899-8628-1ef9371ec9a2" (UID: "277fa9ad-963b-4899-8628-1ef9371ec9a2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:56:56.346104 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.346053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277fa9ad-963b-4899-8628-1ef9371ec9a2-kube-api-access-m8k7m" (OuterVolumeSpecName: "kube-api-access-m8k7m") pod "277fa9ad-963b-4899-8628-1ef9371ec9a2" (UID: "277fa9ad-963b-4899-8628-1ef9371ec9a2"). InnerVolumeSpecName "kube-api-access-m8k7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:56:56.445318 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.445289    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-oauth-serving-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:56:56.445318 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.445316    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-config\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:56:56.445476 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.445331    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-oauth-config\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:56:56.445476 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.445345    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:56:56.445476 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.445357    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8k7m\" (UniqueName: \"kubernetes.io/projected/277fa9ad-963b-4899-8628-1ef9371ec9a2-kube-api-access-m8k7m\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:56:56.445476 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.445368    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/277fa9ad-963b-4899-8628-1ef9371ec9a2-console-serving-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:56:56.445476 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:56.445380    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/277fa9ad-963b-4899-8628-1ef9371ec9a2-trusted-ca-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:56:57.030220 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:57.030185    2580 generic.go:356] "Generic (PLEG): container finished" podID="277fa9ad-963b-4899-8628-1ef9371ec9a2" containerID="3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5" exitCode=0
Sep 14 02:56:57.030659 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:57.030227    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656d967696-phm4z"
Sep 14 02:56:57.030659 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:57.030268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656d967696-phm4z" event={"ID":"277fa9ad-963b-4899-8628-1ef9371ec9a2","Type":"ContainerDied","Data":"3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5"}
Sep 14 02:56:57.030659 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:57.030332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656d967696-phm4z" event={"ID":"277fa9ad-963b-4899-8628-1ef9371ec9a2","Type":"ContainerDied","Data":"45a3e63c9962d321c5773f3989e963e1ad1754f8288bfc4a80e84975e543f142"}
Sep 14 02:56:57.030659 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:57.030354    2580 scope.go:117] "RemoveContainer" containerID="3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5"
Sep 14 02:56:57.041344 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:57.041322    2580 scope.go:117] "RemoveContainer" containerID="3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5"
Sep 14 02:56:57.041637 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:56:57.041614    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5\": container with ID starting with 3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5 not found: ID does not exist" containerID="3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5"
Sep 14 02:56:57.041730 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:57.041650    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5"} err="failed to get container status \"3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5\": rpc error: code = NotFound desc = could not find container \"3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5\": container with ID starting with 3eefe5dd165ffa43a8b0dc2bb5547d5a09a6c622c141906729bfceec3bd0dbc5 not found: ID does not exist"
Sep 14 02:56:57.052529 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:57.052510    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-656d967696-phm4z"]
Sep 14 02:56:57.054813 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:57.054790    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-656d967696-phm4z"]
Sep 14 02:56:57.757197 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:56:57.757161    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="277fa9ad-963b-4899-8628-1ef9371ec9a2" path="/var/lib/kubelet/pods/277fa9ad-963b-4899-8628-1ef9371ec9a2/volumes"
Sep 14 02:57:01.050247 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:01.050195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s" event={"ID":"b3dc5e8a-6ece-4805-b005-0f8e70ff1955","Type":"ContainerStarted","Data":"04ba08ae5fbaeb9b70df9a55f6fd283f575ffb94b90f65d59a3c29e0b6895ade"}
Sep 14 02:57:02.055002 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:02.054967    2580 generic.go:356] "Generic (PLEG): container finished" podID="b3dc5e8a-6ece-4805-b005-0f8e70ff1955" containerID="04ba08ae5fbaeb9b70df9a55f6fd283f575ffb94b90f65d59a3c29e0b6895ade" exitCode=0
Sep 14 02:57:02.055442 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:02.055009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s" event={"ID":"b3dc5e8a-6ece-4805-b005-0f8e70ff1955","Type":"ContainerDied","Data":"04ba08ae5fbaeb9b70df9a55f6fd283f575ffb94b90f65d59a3c29e0b6895ade"}
Sep 14 02:57:05.067848 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:05.067806    2580 generic.go:356] "Generic (PLEG): container finished" podID="b3dc5e8a-6ece-4805-b005-0f8e70ff1955" containerID="6ae080add7cab1433bc12f0baccf11b75332a389327e68885909d45773551a6b" exitCode=0
Sep 14 02:57:05.068222 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:05.067898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s" event={"ID":"b3dc5e8a-6ece-4805-b005-0f8e70ff1955","Type":"ContainerDied","Data":"6ae080add7cab1433bc12f0baccf11b75332a389327e68885909d45773551a6b"}
Sep 14 02:57:12.103728 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:12.103654    2580 generic.go:356] "Generic (PLEG): container finished" podID="b3dc5e8a-6ece-4805-b005-0f8e70ff1955" containerID="e228476a4646544f07944347db6f15be09cd5560a59b6db518c07e11c2e6bf41" exitCode=0
Sep 14 02:57:12.103728 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:12.103713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s" event={"ID":"b3dc5e8a-6ece-4805-b005-0f8e70ff1955","Type":"ContainerDied","Data":"e228476a4646544f07944347db6f15be09cd5560a59b6db518c07e11c2e6bf41"}
Sep 14 02:57:13.225132 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.225110    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:57:13.377171 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.377106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-bundle\") pod \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") "
Sep 14 02:57:13.377171 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.377158    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-util\") pod \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") "
Sep 14 02:57:13.377338 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.377180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbdlf\" (UniqueName: \"kubernetes.io/projected/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-kube-api-access-fbdlf\") pod \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\" (UID: \"b3dc5e8a-6ece-4805-b005-0f8e70ff1955\") "
Sep 14 02:57:13.377681 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.377655    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-bundle" (OuterVolumeSpecName: "bundle") pod "b3dc5e8a-6ece-4805-b005-0f8e70ff1955" (UID: "b3dc5e8a-6ece-4805-b005-0f8e70ff1955"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:57:13.379306 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.379276    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-kube-api-access-fbdlf" (OuterVolumeSpecName: "kube-api-access-fbdlf") pod "b3dc5e8a-6ece-4805-b005-0f8e70ff1955" (UID: "b3dc5e8a-6ece-4805-b005-0f8e70ff1955"). InnerVolumeSpecName "kube-api-access-fbdlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:57:13.381404 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.381381    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-util" (OuterVolumeSpecName: "util") pod "b3dc5e8a-6ece-4805-b005-0f8e70ff1955" (UID: "b3dc5e8a-6ece-4805-b005-0f8e70ff1955"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:57:13.477964 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.477943    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-util\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:13.477964 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.477962    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbdlf\" (UniqueName: \"kubernetes.io/projected/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-kube-api-access-fbdlf\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:13.478109 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.477974    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3dc5e8a-6ece-4805-b005-0f8e70ff1955-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:13.667168 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.667100    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 02:57:13.672423 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.672403    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 02:57:13.682438 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:13.682412    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 14 02:57:14.112360 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:14.112331    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s"
Sep 14 02:57:14.112360 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:14.112345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r5v6s" event={"ID":"b3dc5e8a-6ece-4805-b005-0f8e70ff1955","Type":"ContainerDied","Data":"8f3669a9727fb7727ba9050b8c82751e8cb11d837c0b68c9dd671bab23345f38"}
Sep 14 02:57:14.112610 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:14.112396    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f3669a9727fb7727ba9050b8c82751e8cb11d837c0b68c9dd671bab23345f38"
Sep 14 02:57:17.588957 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.588920    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h"]
Sep 14 02:57:17.589338 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.589232    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3dc5e8a-6ece-4805-b005-0f8e70ff1955" containerName="util"
Sep 14 02:57:17.589338 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.589249    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3dc5e8a-6ece-4805-b005-0f8e70ff1955" containerName="util"
Sep 14 02:57:17.589338 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.589262    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="277fa9ad-963b-4899-8628-1ef9371ec9a2" containerName="console"
Sep 14 02:57:17.589338 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.589268    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="277fa9ad-963b-4899-8628-1ef9371ec9a2" containerName="console"
Sep 14 02:57:17.589338 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.589279    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3dc5e8a-6ece-4805-b005-0f8e70ff1955" containerName="pull"
Sep 14 02:57:17.589338 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.589285    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3dc5e8a-6ece-4805-b005-0f8e70ff1955" containerName="pull"
Sep 14 02:57:17.589338 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.589293    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3dc5e8a-6ece-4805-b005-0f8e70ff1955" containerName="extract"
Sep 14 02:57:17.589338 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.589298    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3dc5e8a-6ece-4805-b005-0f8e70ff1955" containerName="extract"
Sep 14 02:57:17.589576 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.589357    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="277fa9ad-963b-4899-8628-1ef9371ec9a2" containerName="console"
Sep 14 02:57:17.589576 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.589365    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3dc5e8a-6ece-4805-b005-0f8e70ff1955" containerName="extract"
Sep 14 02:57:17.639373 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.639349    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h"]
Sep 14 02:57:17.639509 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.639443    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h"
Sep 14 02:57:17.641753 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.641723    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 14 02:57:17.641753 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.641736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7bgkq\""
Sep 14 02:57:17.641999 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.641985    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 14 02:57:17.809374 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.809343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5fvg\" (UniqueName: \"kubernetes.io/projected/dcae75dd-1409-4012-8c82-de00db936f9b-kube-api-access-j5fvg\") pod \"cert-manager-operator-controller-manager-55c444bf86-4xh7h\" (UID: \"dcae75dd-1409-4012-8c82-de00db936f9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h"
Sep 14 02:57:17.809506 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.809386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcae75dd-1409-4012-8c82-de00db936f9b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4xh7h\" (UID: \"dcae75dd-1409-4012-8c82-de00db936f9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h"
Sep 14 02:57:17.909926 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.909868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcae75dd-1409-4012-8c82-de00db936f9b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4xh7h\" (UID: \"dcae75dd-1409-4012-8c82-de00db936f9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h"
Sep 14 02:57:17.910045 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.909927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5fvg\" (UniqueName: \"kubernetes.io/projected/dcae75dd-1409-4012-8c82-de00db936f9b-kube-api-access-j5fvg\") pod \"cert-manager-operator-controller-manager-55c444bf86-4xh7h\" (UID: \"dcae75dd-1409-4012-8c82-de00db936f9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h"
Sep 14 02:57:17.910244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.910227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcae75dd-1409-4012-8c82-de00db936f9b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4xh7h\" (UID: \"dcae75dd-1409-4012-8c82-de00db936f9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h"
Sep 14 02:57:17.919096 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.919077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5fvg\" (UniqueName: \"kubernetes.io/projected/dcae75dd-1409-4012-8c82-de00db936f9b-kube-api-access-j5fvg\") pod \"cert-manager-operator-controller-manager-55c444bf86-4xh7h\" (UID: \"dcae75dd-1409-4012-8c82-de00db936f9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h"
Sep 14 02:57:17.948392 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:17.948369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h"
Sep 14 02:57:18.240703 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:18.240680    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h"]
Sep 14 02:57:18.242593 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:57:18.242567    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcae75dd_1409_4012_8c82_de00db936f9b.slice/crio-189e3267e736fd1403577ddbb595db3178bf6aa940187abe0de6d41e49d1b1b0 WatchSource:0}: Error finding container 189e3267e736fd1403577ddbb595db3178bf6aa940187abe0de6d41e49d1b1b0: Status 404 returned error can't find the container with id 189e3267e736fd1403577ddbb595db3178bf6aa940187abe0de6d41e49d1b1b0
Sep 14 02:57:18.245489 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:18.245464    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 02:57:19.131784 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:19.131739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h" event={"ID":"dcae75dd-1409-4012-8c82-de00db936f9b","Type":"ContainerStarted","Data":"189e3267e736fd1403577ddbb595db3178bf6aa940187abe0de6d41e49d1b1b0"}
Sep 14 02:57:21.145294 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:21.145262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h" event={"ID":"dcae75dd-1409-4012-8c82-de00db936f9b","Type":"ContainerStarted","Data":"e95af82b25bde0bd526436873e962355b1005cf3d19010050d7261d11bee08ba"}
Sep 14 02:57:21.167766 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:21.167722    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4xh7h" podStartSLOduration=2.234811488 podStartE2EDuration="4.167708046s" totalImagesPullingTime="1.932896558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:57:21.165199197 +0000 UTC m=+307.891157250" watchObservedRunningTime="2026-09-14 02:57:21.167708046 +0000 UTC m=+307.893666097"
Sep 14 02:57:22.005865 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.005830    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"]
Sep 14 02:57:22.008084 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.008065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:22.011130 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.011112    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c2rkr\""
Sep 14 02:57:22.011418 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.011401    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 02:57:22.011632 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.011616    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 02:57:22.026951 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.026932    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"]
Sep 14 02:57:22.036806 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.036773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pw9q\" (UniqueName: \"kubernetes.io/projected/a3d0121f-6da8-4275-84ff-69a726e49c01-kube-api-access-4pw9q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:22.036940 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.036865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:22.036940 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.036911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:22.138187 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.138159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pw9q\" (UniqueName: \"kubernetes.io/projected/a3d0121f-6da8-4275-84ff-69a726e49c01-kube-api-access-4pw9q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:22.138329 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.138211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:22.138329 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.138249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:22.138667 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.138646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:22.138718 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.138656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:22.150468 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.150439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pw9q\" (UniqueName: \"kubernetes.io/projected/a3d0121f-6da8-4275-84ff-69a726e49c01-kube-api-access-4pw9q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:22.316764 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.316701    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:22.614442 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:22.614415    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"]
Sep 14 02:57:22.616898 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:57:22.616867    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d0121f_6da8_4275_84ff_69a726e49c01.slice/crio-eb9353cf66caee8f97d0e48bc704c1d31022e3c57d7c222fbc8aa95d8990b227 WatchSource:0}: Error finding container eb9353cf66caee8f97d0e48bc704c1d31022e3c57d7c222fbc8aa95d8990b227: Status 404 returned error can't find the container with id eb9353cf66caee8f97d0e48bc704c1d31022e3c57d7c222fbc8aa95d8990b227
Sep 14 02:57:23.156174 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:23.156146    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3d0121f-6da8-4275-84ff-69a726e49c01" containerID="97c816b8d2af51c326f16d5c5e4758d2357d757bd23f746ab1043f2db0bd298e" exitCode=0
Sep 14 02:57:23.156529 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:23.156231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d" event={"ID":"a3d0121f-6da8-4275-84ff-69a726e49c01","Type":"ContainerDied","Data":"97c816b8d2af51c326f16d5c5e4758d2357d757bd23f746ab1043f2db0bd298e"}
Sep 14 02:57:23.156529 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:23.156279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d" event={"ID":"a3d0121f-6da8-4275-84ff-69a726e49c01","Type":"ContainerStarted","Data":"eb9353cf66caee8f97d0e48bc704c1d31022e3c57d7c222fbc8aa95d8990b227"}
Sep 14 02:57:25.166820 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:25.166787    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3d0121f-6da8-4275-84ff-69a726e49c01" containerID="b004c396b793fabb209ab60addde93cfa087fb1e5fb1315d0949064bb4d5c988" exitCode=0
Sep 14 02:57:25.167183 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:25.166867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d" event={"ID":"a3d0121f-6da8-4275-84ff-69a726e49c01","Type":"ContainerDied","Data":"b004c396b793fabb209ab60addde93cfa087fb1e5fb1315d0949064bb4d5c988"}
Sep 14 02:57:26.172597 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:26.172568    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3d0121f-6da8-4275-84ff-69a726e49c01" containerID="ba312dbb8cdd41cbbd45e87dcaee8496cdcc4970efc2a56cb9dc60140b017ace" exitCode=0
Sep 14 02:57:26.172974 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:26.172641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d" event={"ID":"a3d0121f-6da8-4275-84ff-69a726e49c01","Type":"ContainerDied","Data":"ba312dbb8cdd41cbbd45e87dcaee8496cdcc4970efc2a56cb9dc60140b017ace"}
Sep 14 02:57:27.296732 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:27.296710    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:27.377315 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:27.377287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-util\") pod \"a3d0121f-6da8-4275-84ff-69a726e49c01\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") "
Sep 14 02:57:27.377433 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:27.377323    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-bundle\") pod \"a3d0121f-6da8-4275-84ff-69a726e49c01\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") "
Sep 14 02:57:27.377433 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:27.377377    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pw9q\" (UniqueName: \"kubernetes.io/projected/a3d0121f-6da8-4275-84ff-69a726e49c01-kube-api-access-4pw9q\") pod \"a3d0121f-6da8-4275-84ff-69a726e49c01\" (UID: \"a3d0121f-6da8-4275-84ff-69a726e49c01\") "
Sep 14 02:57:27.377800 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:27.377767    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-bundle" (OuterVolumeSpecName: "bundle") pod "a3d0121f-6da8-4275-84ff-69a726e49c01" (UID: "a3d0121f-6da8-4275-84ff-69a726e49c01"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:57:27.379426 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:27.379397    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d0121f-6da8-4275-84ff-69a726e49c01-kube-api-access-4pw9q" (OuterVolumeSpecName: "kube-api-access-4pw9q") pod "a3d0121f-6da8-4275-84ff-69a726e49c01" (UID: "a3d0121f-6da8-4275-84ff-69a726e49c01"). InnerVolumeSpecName "kube-api-access-4pw9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:57:27.382476 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:27.382443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-util" (OuterVolumeSpecName: "util") pod "a3d0121f-6da8-4275-84ff-69a726e49c01" (UID: "a3d0121f-6da8-4275-84ff-69a726e49c01"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:57:27.477950 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:27.477884    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pw9q\" (UniqueName: \"kubernetes.io/projected/a3d0121f-6da8-4275-84ff-69a726e49c01-kube-api-access-4pw9q\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:27.477950 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:27.477911    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-util\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:27.477950 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:27.477920    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3d0121f-6da8-4275-84ff-69a726e49c01-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:28.181451 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:28.181364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d" event={"ID":"a3d0121f-6da8-4275-84ff-69a726e49c01","Type":"ContainerDied","Data":"eb9353cf66caee8f97d0e48bc704c1d31022e3c57d7c222fbc8aa95d8990b227"}
Sep 14 02:57:28.181451 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:28.181396    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwb7d"
Sep 14 02:57:28.181451 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:28.181409    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb9353cf66caee8f97d0e48bc704c1d31022e3c57d7c222fbc8aa95d8990b227"
Sep 14 02:57:31.719168 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.719137    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4"]
Sep 14 02:57:31.719546 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.719420    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d0121f-6da8-4275-84ff-69a726e49c01" containerName="extract"
Sep 14 02:57:31.719546 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.719431    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d0121f-6da8-4275-84ff-69a726e49c01" containerName="extract"
Sep 14 02:57:31.719546 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.719439    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d0121f-6da8-4275-84ff-69a726e49c01" containerName="pull"
Sep 14 02:57:31.719546 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.719445    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d0121f-6da8-4275-84ff-69a726e49c01" containerName="pull"
Sep 14 02:57:31.719546 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.719452    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d0121f-6da8-4275-84ff-69a726e49c01" containerName="util"
Sep 14 02:57:31.719546 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.719457    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d0121f-6da8-4275-84ff-69a726e49c01" containerName="util"
Sep 14 02:57:31.719546 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.719497    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d0121f-6da8-4275-84ff-69a726e49c01" containerName="extract"
Sep 14 02:57:31.721359 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.721343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4"
Sep 14 02:57:31.723529 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.723507    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 14 02:57:31.724064 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.724043    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 14 02:57:31.724157 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.724050    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fqgm8\""
Sep 14 02:57:31.731567 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.731546    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4"]
Sep 14 02:57:31.807288 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.807260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96c350f8-214a-4c63-bfed-89d6929be48a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nf6l4\" (UID: \"96c350f8-214a-4c63-bfed-89d6929be48a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4"
Sep 14 02:57:31.807288 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.807289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84lv4\" (UniqueName: \"kubernetes.io/projected/96c350f8-214a-4c63-bfed-89d6929be48a-kube-api-access-84lv4\") pod \"openshift-lws-operator-fd8ccff4c-nf6l4\" (UID: \"96c350f8-214a-4c63-bfed-89d6929be48a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4"
Sep 14 02:57:31.907918 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.907889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96c350f8-214a-4c63-bfed-89d6929be48a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nf6l4\" (UID: \"96c350f8-214a-4c63-bfed-89d6929be48a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4"
Sep 14 02:57:31.907918 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.907920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lv4\" (UniqueName: \"kubernetes.io/projected/96c350f8-214a-4c63-bfed-89d6929be48a-kube-api-access-84lv4\") pod \"openshift-lws-operator-fd8ccff4c-nf6l4\" (UID: \"96c350f8-214a-4c63-bfed-89d6929be48a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4"
Sep 14 02:57:31.908323 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.908306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96c350f8-214a-4c63-bfed-89d6929be48a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nf6l4\" (UID: \"96c350f8-214a-4c63-bfed-89d6929be48a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4"
Sep 14 02:57:31.921308 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:31.921280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84lv4\" (UniqueName: \"kubernetes.io/projected/96c350f8-214a-4c63-bfed-89d6929be48a-kube-api-access-84lv4\") pod \"openshift-lws-operator-fd8ccff4c-nf6l4\" (UID: \"96c350f8-214a-4c63-bfed-89d6929be48a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4"
Sep 14 02:57:32.029928 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:32.029902    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4"
Sep 14 02:57:32.298697 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:32.298629    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4"]
Sep 14 02:57:32.301842 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:57:32.301816    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96c350f8_214a_4c63_bfed_89d6929be48a.slice/crio-602bf1b9de02299ae46d79c0d01bb72fc373579434c35629c2be33009c7856ca WatchSource:0}: Error finding container 602bf1b9de02299ae46d79c0d01bb72fc373579434c35629c2be33009c7856ca: Status 404 returned error can't find the container with id 602bf1b9de02299ae46d79c0d01bb72fc373579434c35629c2be33009c7856ca
Sep 14 02:57:33.202415 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:33.202384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4" event={"ID":"96c350f8-214a-4c63-bfed-89d6929be48a","Type":"ContainerStarted","Data":"602bf1b9de02299ae46d79c0d01bb72fc373579434c35629c2be33009c7856ca"}
Sep 14 02:57:35.211736 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:35.211695    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4" event={"ID":"96c350f8-214a-4c63-bfed-89d6929be48a","Type":"ContainerStarted","Data":"1a9b3ff9f96ef1ee1b435d8568e8e90d69a185b17dfb483d6264437907ce1928"}
Sep 14 02:57:35.256312 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:35.254491    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nf6l4" podStartSLOduration=1.993419364 podStartE2EDuration="4.254470715s" totalImagesPullingTime="2.261051351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:57:35.252840532 +0000 UTC m=+321.978798583" watchObservedRunningTime="2026-09-14 02:57:35.254470715 +0000 UTC m=+321.980428769"
Sep 14 02:57:37.499049 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.498996    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"]
Sep 14 02:57:37.501244 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.501229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:37.503622 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.503598    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 02:57:37.503751 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.503631    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 02:57:37.504137 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.504118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c2rkr\""
Sep 14 02:57:37.511431 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.511409    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"]
Sep 14 02:57:37.546130 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.546104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:37.546242 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.546137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:37.546242 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.546169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjv2\" (UniqueName: \"kubernetes.io/projected/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-kube-api-access-wsjv2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:37.646704 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.646674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:37.646816 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.646715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:37.646816 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.646761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjv2\" (UniqueName: \"kubernetes.io/projected/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-kube-api-access-wsjv2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:37.647050 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.647004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:37.647121 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.647095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:37.655756 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.655736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjv2\" (UniqueName: \"kubernetes.io/projected/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-kube-api-access-wsjv2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:37.810831 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:37.810809    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:38.078901 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:38.078866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"]
Sep 14 02:57:38.081486 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:57:38.081457    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec610a6_e8a8_48f5_b696_17b5d9c6a37e.slice/crio-175795098ea00a6a33933852b8fc0f4754ccea0af8558446ca025eae3d73adce WatchSource:0}: Error finding container 175795098ea00a6a33933852b8fc0f4754ccea0af8558446ca025eae3d73adce: Status 404 returned error can't find the container with id 175795098ea00a6a33933852b8fc0f4754ccea0af8558446ca025eae3d73adce
Sep 14 02:57:38.226379 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:38.226345    2580 generic.go:356] "Generic (PLEG): container finished" podID="2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" containerID="cdc11827b28ba2b29ac0e09909c6397701506711aea18e76477e44b99e93d13b" exitCode=0
Sep 14 02:57:38.226501 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:38.226431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc" event={"ID":"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e","Type":"ContainerDied","Data":"cdc11827b28ba2b29ac0e09909c6397701506711aea18e76477e44b99e93d13b"}
Sep 14 02:57:38.226501 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:38.226487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc" event={"ID":"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e","Type":"ContainerStarted","Data":"175795098ea00a6a33933852b8fc0f4754ccea0af8558446ca025eae3d73adce"}
Sep 14 02:57:39.232560 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:39.232481    2580 generic.go:356] "Generic (PLEG): container finished" podID="2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" containerID="cd05d1447a08c6acb236c623b9f53254e0f825850c1b2b690d7e21ab1fadcc75" exitCode=0
Sep 14 02:57:39.232560 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:39.232534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc" event={"ID":"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e","Type":"ContainerDied","Data":"cd05d1447a08c6acb236c623b9f53254e0f825850c1b2b690d7e21ab1fadcc75"}
Sep 14 02:57:40.238982 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:40.238946    2580 generic.go:356] "Generic (PLEG): container finished" podID="2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" containerID="b2d278bf80919b2192e7c3a6295b0ce5ebc393557da7b48edb2e67c7fddbffb7" exitCode=0
Sep 14 02:57:40.239358 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:40.238989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc" event={"ID":"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e","Type":"ContainerDied","Data":"b2d278bf80919b2192e7c3a6295b0ce5ebc393557da7b48edb2e67c7fddbffb7"}
Sep 14 02:57:41.359998 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:41.359976    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:41.478726 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:41.478697    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsjv2\" (UniqueName: \"kubernetes.io/projected/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-kube-api-access-wsjv2\") pod \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") "
Sep 14 02:57:41.478898 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:41.478758    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-util\") pod \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") "
Sep 14 02:57:41.478898 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:41.478787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-bundle\") pod \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\" (UID: \"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e\") "
Sep 14 02:57:41.479514 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:41.479486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-bundle" (OuterVolumeSpecName: "bundle") pod "2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" (UID: "2ec610a6-e8a8-48f5-b696-17b5d9c6a37e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:57:41.480873 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:41.480844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-kube-api-access-wsjv2" (OuterVolumeSpecName: "kube-api-access-wsjv2") pod "2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" (UID: "2ec610a6-e8a8-48f5-b696-17b5d9c6a37e"). InnerVolumeSpecName "kube-api-access-wsjv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:57:41.483984 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:41.483940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-util" (OuterVolumeSpecName: "util") pod "2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" (UID: "2ec610a6-e8a8-48f5-b696-17b5d9c6a37e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:57:41.579167 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:41.579140    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:41.579167 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:41.579163    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsjv2\" (UniqueName: \"kubernetes.io/projected/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-kube-api-access-wsjv2\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:41.579167 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:41.579172    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ec610a6-e8a8-48f5-b696-17b5d9c6a37e-util\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:42.249629 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:42.249603    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc"
Sep 14 02:57:42.249629 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:42.249605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflhphc" event={"ID":"2ec610a6-e8a8-48f5-b696-17b5d9c6a37e","Type":"ContainerDied","Data":"175795098ea00a6a33933852b8fc0f4754ccea0af8558446ca025eae3d73adce"}
Sep 14 02:57:42.249813 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:42.249658    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="175795098ea00a6a33933852b8fc0f4754ccea0af8558446ca025eae3d73adce"
Sep 14 02:57:48.122301 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.122266    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"]
Sep 14 02:57:48.122844 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.122643    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" containerName="pull"
Sep 14 02:57:48.122844 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.122661    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" containerName="pull"
Sep 14 02:57:48.122844 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.122675    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" containerName="util"
Sep 14 02:57:48.122844 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.122684    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" containerName="util"
Sep 14 02:57:48.122844 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.122711    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" containerName="extract"
Sep 14 02:57:48.122844 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.122719    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" containerName="extract"
Sep 14 02:57:48.122844 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.122817    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ec610a6-e8a8-48f5-b696-17b5d9c6a37e" containerName="extract"
Sep 14 02:57:48.126348 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.126325    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:48.128800 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.128775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 02:57:48.128905 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.128797    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 02:57:48.129362 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.129341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c2rkr\""
Sep 14 02:57:48.139425 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.139404    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"]
Sep 14 02:57:48.224757 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.224735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2778\" (UniqueName: \"kubernetes.io/projected/16dc5f4a-4ff2-4ace-9307-194191a7adbc-kube-api-access-f2778\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:48.224885 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.224775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:48.224885 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.224861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:48.325732 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.325705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:48.325831 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.325745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2778\" (UniqueName: \"kubernetes.io/projected/16dc5f4a-4ff2-4ace-9307-194191a7adbc-kube-api-access-f2778\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:48.325886 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.325867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:48.326099 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.326081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:48.326174 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.326158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:48.333854 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.333834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2778\" (UniqueName: \"kubernetes.io/projected/16dc5f4a-4ff2-4ace-9307-194191a7adbc-kube-api-access-f2778\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:48.434541 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.434489    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:48.706095 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:48.706068    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"]
Sep 14 02:57:48.708249 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:57:48.708212    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16dc5f4a_4ff2_4ace_9307_194191a7adbc.slice/crio-450dae2e906acee4446c21b71b2df6a15b7cf2790b225f0733c7a0d77ddf9417 WatchSource:0}: Error finding container 450dae2e906acee4446c21b71b2df6a15b7cf2790b225f0733c7a0d77ddf9417: Status 404 returned error can't find the container with id 450dae2e906acee4446c21b71b2df6a15b7cf2790b225f0733c7a0d77ddf9417
Sep 14 02:57:49.278703 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:49.278670    2580 generic.go:356] "Generic (PLEG): container finished" podID="16dc5f4a-4ff2-4ace-9307-194191a7adbc" containerID="e8a0b3819690506f8dfbcf9e6b9abedb3f5c347e0c733a0add26488e6d05b7b2" exitCode=0
Sep 14 02:57:49.278703 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:49.278707    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr" event={"ID":"16dc5f4a-4ff2-4ace-9307-194191a7adbc","Type":"ContainerDied","Data":"e8a0b3819690506f8dfbcf9e6b9abedb3f5c347e0c733a0add26488e6d05b7b2"}
Sep 14 02:57:49.279115 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:49.278739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr" event={"ID":"16dc5f4a-4ff2-4ace-9307-194191a7adbc","Type":"ContainerStarted","Data":"450dae2e906acee4446c21b71b2df6a15b7cf2790b225f0733c7a0d77ddf9417"}
Sep 14 02:57:50.285352 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.285309    2580 generic.go:356] "Generic (PLEG): container finished" podID="16dc5f4a-4ff2-4ace-9307-194191a7adbc" containerID="2b66d120b5667680444ceb92536bbb893939ed8ad6460bb4a537c0f874e92575" exitCode=0
Sep 14 02:57:50.285719 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.285355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr" event={"ID":"16dc5f4a-4ff2-4ace-9307-194191a7adbc","Type":"ContainerDied","Data":"2b66d120b5667680444ceb92536bbb893939ed8ad6460bb4a537c0f874e92575"}
Sep 14 02:57:50.467900 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.467872    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"]
Sep 14 02:57:50.470915 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.470900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.474191 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.474173    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 14 02:57:50.474935 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.474916    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 14 02:57:50.474980 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.474930    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-49p7j\""
Sep 14 02:57:50.474980 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.474959    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 14 02:57:50.475090 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.475019    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 14 02:57:50.491968 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.491945    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"]
Sep 14 02:57:50.542916 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.542861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.542916 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.542894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.542916 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.542913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gjj\" (UniqueName: \"kubernetes.io/projected/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-kube-api-access-69gjj\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.543134 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.542931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-tmp\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.543134 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.543015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-manifests\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.643618 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.643593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-manifests\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.643750 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.643649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.643750 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.643672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.643750 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.643688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gjj\" (UniqueName: \"kubernetes.io/projected/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-kube-api-access-69gjj\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.643750 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.643707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-tmp\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.644005 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.643982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-manifests\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.645836 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.645807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-tmp\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.646077 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.646061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.646246 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.646228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.659361 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.659338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gjj\" (UniqueName: \"kubernetes.io/projected/6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e-kube-api-access-69gjj\") pod \"opendatahub-operator-controller-manager-67ffb7699c-crjkz\" (UID: \"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:50.803961 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:50.803867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:51.095451 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:51.095414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"]
Sep 14 02:57:51.096005 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:57:51.095973    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3f0a6c_9145_49f1_b4e6_a2ead4d3f75e.slice/crio-bd45f3556c98546599982fa4c10ea92dc1a298b123ea275b5433a11fb6d0aab7 WatchSource:0}: Error finding container bd45f3556c98546599982fa4c10ea92dc1a298b123ea275b5433a11fb6d0aab7: Status 404 returned error can't find the container with id bd45f3556c98546599982fa4c10ea92dc1a298b123ea275b5433a11fb6d0aab7
Sep 14 02:57:51.291974 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:51.291943    2580 generic.go:356] "Generic (PLEG): container finished" podID="16dc5f4a-4ff2-4ace-9307-194191a7adbc" containerID="3a066d2ebebcb49d571caf6bcbab5e8079ada087ad3e80d7a5118dcee21c111b" exitCode=0
Sep 14 02:57:51.292412 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:51.292017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr" event={"ID":"16dc5f4a-4ff2-4ace-9307-194191a7adbc","Type":"ContainerDied","Data":"3a066d2ebebcb49d571caf6bcbab5e8079ada087ad3e80d7a5118dcee21c111b"}
Sep 14 02:57:51.293293 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:51.293273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz" event={"ID":"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e","Type":"ContainerStarted","Data":"bd45f3556c98546599982fa4c10ea92dc1a298b123ea275b5433a11fb6d0aab7"}
Sep 14 02:57:53.998079 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:53.998053    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:54.074603 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.074582    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-util\") pod \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") "
Sep 14 02:57:54.074688 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.074629    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2778\" (UniqueName: \"kubernetes.io/projected/16dc5f4a-4ff2-4ace-9307-194191a7adbc-kube-api-access-f2778\") pod \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") "
Sep 14 02:57:54.074688 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.074681    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-bundle\") pod \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\" (UID: \"16dc5f4a-4ff2-4ace-9307-194191a7adbc\") "
Sep 14 02:57:54.075355 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.075330    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-bundle" (OuterVolumeSpecName: "bundle") pod "16dc5f4a-4ff2-4ace-9307-194191a7adbc" (UID: "16dc5f4a-4ff2-4ace-9307-194191a7adbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:57:54.076680 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.076655    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16dc5f4a-4ff2-4ace-9307-194191a7adbc-kube-api-access-f2778" (OuterVolumeSpecName: "kube-api-access-f2778") pod "16dc5f4a-4ff2-4ace-9307-194191a7adbc" (UID: "16dc5f4a-4ff2-4ace-9307-194191a7adbc"). InnerVolumeSpecName "kube-api-access-f2778". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:57:54.078558 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.078534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-util" (OuterVolumeSpecName: "util") pod "16dc5f4a-4ff2-4ace-9307-194191a7adbc" (UID: "16dc5f4a-4ff2-4ace-9307-194191a7adbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:57:54.175757 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.175731    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:54.175757 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.175753    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16dc5f4a-4ff2-4ace-9307-194191a7adbc-util\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:54.175757 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.175763    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2778\" (UniqueName: \"kubernetes.io/projected/16dc5f4a-4ff2-4ace-9307-194191a7adbc-kube-api-access-f2778\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:57:54.309520 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.309458    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr"
Sep 14 02:57:54.309520 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.309490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hbtbr" event={"ID":"16dc5f4a-4ff2-4ace-9307-194191a7adbc","Type":"ContainerDied","Data":"450dae2e906acee4446c21b71b2df6a15b7cf2790b225f0733c7a0d77ddf9417"}
Sep 14 02:57:54.309748 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.309539    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="450dae2e906acee4446c21b71b2df6a15b7cf2790b225f0733c7a0d77ddf9417"
Sep 14 02:57:54.311142 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:54.311117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz" event={"ID":"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e","Type":"ContainerStarted","Data":"8a6eb769cc262fa71cadc161f6d754bbe0c4bf81600f2bee5036081e95c68b3c"}
Sep 14 02:57:55.316766 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:55.316734    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e" containerID="8a6eb769cc262fa71cadc161f6d754bbe0c4bf81600f2bee5036081e95c68b3c" exitCode=0
Sep 14 02:57:55.317213 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:55.316821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz" event={"ID":"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e","Type":"ContainerDied","Data":"8a6eb769cc262fa71cadc161f6d754bbe0c4bf81600f2bee5036081e95c68b3c"}
Sep 14 02:57:56.322807 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:56.322769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz" event={"ID":"6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e","Type":"ContainerStarted","Data":"d2ac83258fde70f18f09140cd52fff9085a789d119757d0da0adf602678aa478"}
Sep 14 02:57:56.323177 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:56.322886    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:57:56.352672 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:57:56.352629    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz" podStartSLOduration=2.050010594 podStartE2EDuration="6.352614197s" totalImagesPullingTime="3.314294867s" totalInitContainerRuntime="988.308736ms" isStatefulPod=false podCreationTimestamp="2026-09-14 02:57:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:57:56.349634811 +0000 UTC m=+343.075592862" watchObservedRunningTime="2026-09-14 02:57:56.352614197 +0000 UTC m=+343.078572248"
Sep 14 02:58:07.329104 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:07.329012    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-crjkz"
Sep 14 02:58:08.546188 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.546152    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"]
Sep 14 02:58:08.546548 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.546441    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16dc5f4a-4ff2-4ace-9307-194191a7adbc" containerName="extract"
Sep 14 02:58:08.546548 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.546453    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="16dc5f4a-4ff2-4ace-9307-194191a7adbc" containerName="extract"
Sep 14 02:58:08.546548 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.546467    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16dc5f4a-4ff2-4ace-9307-194191a7adbc" containerName="pull"
Sep 14 02:58:08.546548 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.546476    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="16dc5f4a-4ff2-4ace-9307-194191a7adbc" containerName="pull"
Sep 14 02:58:08.546548 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.546491    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16dc5f4a-4ff2-4ace-9307-194191a7adbc" containerName="util"
Sep 14 02:58:08.546548 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.546496    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="16dc5f4a-4ff2-4ace-9307-194191a7adbc" containerName="util"
Sep 14 02:58:08.546548 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.546542    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="16dc5f4a-4ff2-4ace-9307-194191a7adbc" containerName="extract"
Sep 14 02:58:08.549497 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.549481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.552651 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.552628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mqbb7\""
Sep 14 02:58:08.552651 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.552634    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 14 02:58:08.552850 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.552753    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 14 02:58:08.552850 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.552755    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 14 02:58:08.576047 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.576001    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"]
Sep 14 02:58:08.675139 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.675113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-manager-config\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.675314 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.675158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-cert\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.675314 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.675258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4hml\" (UniqueName: \"kubernetes.io/projected/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-kube-api-access-m4hml\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.675314 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.675298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-metrics-cert\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.776461 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.776436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4hml\" (UniqueName: \"kubernetes.io/projected/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-kube-api-access-m4hml\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.776673 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.776470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-metrics-cert\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.776673 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.776499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-manager-config\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.776673 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.776555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-cert\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.777294 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.777268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-manager-config\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.779054 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.779012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-cert\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.779273 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.779251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-metrics-cert\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.792453 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.792422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4hml\" (UniqueName: \"kubernetes.io/projected/959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a-kube-api-access-m4hml\") pod \"lws-controller-manager-59b9ccfbf8-trr68\" (UID: \"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:08.858680 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:08.858615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:09.162506 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:09.162484    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"]
Sep 14 02:58:09.164665 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:58:09.164642    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959c47ab_a1ab_4a6b_ad95_aeb95acd3e8a.slice/crio-1a9d1554f9551cfbf8a92cd698c71a13561542d6caa2333eb11eb4d24985122c WatchSource:0}: Error finding container 1a9d1554f9551cfbf8a92cd698c71a13561542d6caa2333eb11eb4d24985122c: Status 404 returned error can't find the container with id 1a9d1554f9551cfbf8a92cd698c71a13561542d6caa2333eb11eb4d24985122c
Sep 14 02:58:09.378757 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:09.378720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68" event={"ID":"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a","Type":"ContainerStarted","Data":"1a9d1554f9551cfbf8a92cd698c71a13561542d6caa2333eb11eb4d24985122c"}
Sep 14 02:58:11.388979 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:11.388943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68" event={"ID":"959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a","Type":"ContainerStarted","Data":"aac81769b93b69b2129bf19d20fe04adaddbf7b5a3a6b04875d80d307ebe86f0"}
Sep 14 02:58:11.389426 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:11.389014    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:11.480168 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:11.480098    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68" podStartSLOduration=2.06093717 podStartE2EDuration="3.48008198s" totalImagesPullingTime="1.41914481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:58:11.456399985 +0000 UTC m=+358.182358037" watchObservedRunningTime="2026-09-14 02:58:11.48008198 +0000 UTC m=+358.206040032"
Sep 14 02:58:18.859608 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.859576    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"]
Sep 14 02:58:18.864365 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.864345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:18.867297 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.867271    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mzt2j\""
Sep 14 02:58:18.868550 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.868532    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 14 02:58:18.889908 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.889884    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"]
Sep 14 02:58:18.957041 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.956997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:18.957178 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.957089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fb8508-202c-45bd-a05b-010246a3bec1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:18.957178 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.957126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fb8508-202c-45bd-a05b-010246a3bec1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:18.957178 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.957157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:18.957314 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.957188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fb8508-202c-45bd-a05b-010246a3bec1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:18.957314 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.957215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:18.957314 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.957294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:18.957422 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.957321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:18.957422 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:18.957342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkpsv\" (UniqueName: \"kubernetes.io/projected/29fb8508-202c-45bd-a05b-010246a3bec1-kube-api-access-lkpsv\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.058690 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.058656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.058925 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.058717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fb8508-202c-45bd-a05b-010246a3bec1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.058925 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.058752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fb8508-202c-45bd-a05b-010246a3bec1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.058925 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.058784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.058925 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.058816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fb8508-202c-45bd-a05b-010246a3bec1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.058925 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.058845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.058925 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.058886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.058925 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.058917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.059319 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.058945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkpsv\" (UniqueName: \"kubernetes.io/projected/29fb8508-202c-45bd-a05b-010246a3bec1-kube-api-access-lkpsv\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.059319 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.059093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.059319 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.059207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.059319 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.059302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.059523 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.059344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.059523 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.059459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fb8508-202c-45bd-a05b-010246a3bec1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.061128 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.061097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fb8508-202c-45bd-a05b-010246a3bec1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.061366 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.061348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fb8508-202c-45bd-a05b-010246a3bec1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.076994 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.076974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkpsv\" (UniqueName: \"kubernetes.io/projected/29fb8508-202c-45bd-a05b-010246a3bec1-kube-api-access-lkpsv\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.079401 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.079381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fb8508-202c-45bd-a05b-010246a3bec1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6\" (UID: \"29fb8508-202c-45bd-a05b-010246a3bec1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.173599 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.173546    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:19.454816 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:19.454790    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"]
Sep 14 02:58:19.457152 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:58:19.457125    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fb8508_202c_45bd_a05b_010246a3bec1.slice/crio-5a97365fa9ed4b0740141d6d04cdaf477aac9c53d9d53ee60dd746e49dd90f4c WatchSource:0}: Error finding container 5a97365fa9ed4b0740141d6d04cdaf477aac9c53d9d53ee60dd746e49dd90f4c: Status 404 returned error can't find the container with id 5a97365fa9ed4b0740141d6d04cdaf477aac9c53d9d53ee60dd746e49dd90f4c
Sep 14 02:58:20.437141 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.437058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6" event={"ID":"29fb8508-202c-45bd-a05b-010246a3bec1","Type":"ContainerStarted","Data":"5a97365fa9ed4b0740141d6d04cdaf477aac9c53d9d53ee60dd746e49dd90f4c"}
Sep 14 02:58:20.723771 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.723693    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mmsrq"]
Sep 14 02:58:20.726950 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.726929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq"
Sep 14 02:58:20.733519 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.733498    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 14 02:58:20.734823 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.734805    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hwcwc\""
Sep 14 02:58:20.737603 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.737581    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mmsrq"]
Sep 14 02:58:20.873195 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.873151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49362469-29e3-4602-b79a-96998c748d9b-cert\") pod \"odh-model-controller-7998c67dc4-mmsrq\" (UID: \"49362469-29e3-4602-b79a-96998c748d9b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq"
Sep 14 02:58:20.873360 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.873211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdftk\" (UniqueName: \"kubernetes.io/projected/49362469-29e3-4602-b79a-96998c748d9b-kube-api-access-hdftk\") pod \"odh-model-controller-7998c67dc4-mmsrq\" (UID: \"49362469-29e3-4602-b79a-96998c748d9b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq"
Sep 14 02:58:20.973953 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.973868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49362469-29e3-4602-b79a-96998c748d9b-cert\") pod \"odh-model-controller-7998c67dc4-mmsrq\" (UID: \"49362469-29e3-4602-b79a-96998c748d9b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq"
Sep 14 02:58:20.973953 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.973922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdftk\" (UniqueName: \"kubernetes.io/projected/49362469-29e3-4602-b79a-96998c748d9b-kube-api-access-hdftk\") pod \"odh-model-controller-7998c67dc4-mmsrq\" (UID: \"49362469-29e3-4602-b79a-96998c748d9b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq"
Sep 14 02:58:20.976783 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.976754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49362469-29e3-4602-b79a-96998c748d9b-cert\") pod \"odh-model-controller-7998c67dc4-mmsrq\" (UID: \"49362469-29e3-4602-b79a-96998c748d9b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq"
Sep 14 02:58:20.983629 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:20.983581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdftk\" (UniqueName: \"kubernetes.io/projected/49362469-29e3-4602-b79a-96998c748d9b-kube-api-access-hdftk\") pod \"odh-model-controller-7998c67dc4-mmsrq\" (UID: \"49362469-29e3-4602-b79a-96998c748d9b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq"
Sep 14 02:58:21.036951 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:21.036926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq"
Sep 14 02:58:21.322099 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:21.322065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mmsrq"]
Sep 14 02:58:22.115574 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:58:22.115540    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49362469_29e3_4602_b79a_96998c748d9b.slice/crio-f8cbffd78e7d5f8808f47176f8cf0694a306d6eea54277ee3928f813fbc39d9e WatchSource:0}: Error finding container f8cbffd78e7d5f8808f47176f8cf0694a306d6eea54277ee3928f813fbc39d9e: Status 404 returned error can't find the container with id f8cbffd78e7d5f8808f47176f8cf0694a306d6eea54277ee3928f813fbc39d9e
Sep 14 02:58:22.178563 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:22.178529    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 02:58:22.178672 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:22.178599    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 02:58:22.178672 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:22.178630    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 02:58:22.396235 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:22.396161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-trr68"
Sep 14 02:58:22.452429 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:22.452397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6" event={"ID":"29fb8508-202c-45bd-a05b-010246a3bec1","Type":"ContainerStarted","Data":"cf2eaa8668e349367a567ce0d7d204bd237565c04733a0ded34d924a2759554e"}
Sep 14 02:58:22.453607 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:22.453582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq" event={"ID":"49362469-29e3-4602-b79a-96998c748d9b","Type":"ContainerStarted","Data":"f8cbffd78e7d5f8808f47176f8cf0694a306d6eea54277ee3928f813fbc39d9e"}
Sep 14 02:58:22.514737 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:22.514678    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6" podStartSLOduration=1.7954394740000001 podStartE2EDuration="4.514657678s" totalImagesPullingTime="2.719218204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:58:22.513020447 +0000 UTC m=+369.238978524" watchObservedRunningTime="2026-09-14 02:58:22.514657678 +0000 UTC m=+369.240615731"
Sep 14 02:58:23.174612 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:23.174532    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:23.180305 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:23.180279    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:23.459385 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:23.459309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:23.460389 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:23.460363    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6"
Sep 14 02:58:25.469452 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:25.469365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq" event={"ID":"49362469-29e3-4602-b79a-96998c748d9b","Type":"ContainerStarted","Data":"79153c29afbc13485fd4434747166c0eba4331e4e9c1d458fb1f08a5c2c189d9"}
Sep 14 02:58:25.469872 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:25.469667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq"
Sep 14 02:58:25.490565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:25.490519    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq" podStartSLOduration=2.541571148 podStartE2EDuration="5.490505927s" totalImagesPullingTime="2.948934779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:58:25.489330574 +0000 UTC m=+372.215288637" watchObservedRunningTime="2026-09-14 02:58:25.490505927 +0000 UTC m=+372.216464013"
Sep 14 02:58:29.219873 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.219836    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-666bf7cc8f-v7qng"]
Sep 14 02:58:29.226413 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.226393    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.255899 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.255879    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666bf7cc8f-v7qng"]
Sep 14 02:58:29.342138 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.342114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-console-oauth-config\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.342278 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.342148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-oauth-serving-cert\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.342278 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.342173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-trusted-ca-bundle\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.342278 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.342192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmw4\" (UniqueName: \"kubernetes.io/projected/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-kube-api-access-8nmw4\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.342425 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.342310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-console-config\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.342425 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.342355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-console-serving-cert\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.342425 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.342407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-service-ca\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.443733 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.443708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-console-config\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.443837 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.443741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-console-serving-cert\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.443837 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.443760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-service-ca\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.443927 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.443886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-console-oauth-config\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.443986 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.443928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-oauth-serving-cert\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.444159 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.444137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-trusted-ca-bundle\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.444252 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.444185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmw4\" (UniqueName: \"kubernetes.io/projected/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-kube-api-access-8nmw4\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.444467 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.444442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-service-ca\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.444555 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.444525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-console-config\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.444659 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.444630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-oauth-serving-cert\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.444918 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.444900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-trusted-ca-bundle\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.446253 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.446234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-console-serving-cert\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.446336 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.446277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-console-oauth-config\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.467212 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.467192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmw4\" (UniqueName: \"kubernetes.io/projected/98c6085d-e5f9-43c4-be8f-a0c0a683f91d-kube-api-access-8nmw4\") pod \"console-666bf7cc8f-v7qng\" (UID: \"98c6085d-e5f9-43c4-be8f-a0c0a683f91d\") " pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.535585 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.535558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:29.846719 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:58:29.846690    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c6085d_e5f9_43c4_be8f_a0c0a683f91d.slice/crio-3b53ddc0729c2436efc0af94be1234bca3cbcd2e408bdb984a3f665cccb3e33c WatchSource:0}: Error finding container 3b53ddc0729c2436efc0af94be1234bca3cbcd2e408bdb984a3f665cccb3e33c: Status 404 returned error can't find the container with id 3b53ddc0729c2436efc0af94be1234bca3cbcd2e408bdb984a3f665cccb3e33c
Sep 14 02:58:29.868785 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:29.868750    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666bf7cc8f-v7qng"]
Sep 14 02:58:30.491999 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:30.491963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666bf7cc8f-v7qng" event={"ID":"98c6085d-e5f9-43c4-be8f-a0c0a683f91d","Type":"ContainerStarted","Data":"5118ce6b3ddb590d58ba09cf49282075d4e33c4d75a76cbd565a0a2a22fe8748"}
Sep 14 02:58:30.492347 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:30.492016    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666bf7cc8f-v7qng" event={"ID":"98c6085d-e5f9-43c4-be8f-a0c0a683f91d","Type":"ContainerStarted","Data":"3b53ddc0729c2436efc0af94be1234bca3cbcd2e408bdb984a3f665cccb3e33c"}
Sep 14 02:58:30.570946 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:30.570898    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-666bf7cc8f-v7qng" podStartSLOduration=1.570883008 podStartE2EDuration="1.570883008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:58:30.568751512 +0000 UTC m=+377.294709564" watchObservedRunningTime="2026-09-14 02:58:30.570883008 +0000 UTC m=+377.296841060"
Sep 14 02:58:32.898811 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:32.898777    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"]
Sep 14 02:58:32.901145 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:32.901129    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"
Sep 14 02:58:32.905551 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:32.905531    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 14 02:58:32.905645 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:32.905531    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-f6v6c\""
Sep 14 02:58:32.957581 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:32.957557    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"]
Sep 14 02:58:33.073073 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:33.073026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr85x\" (UniqueName: \"kubernetes.io/projected/a127e29d-b809-4399-a221-c740bfb18d64-kube-api-access-fr85x\") pod \"llmisvc-controller-manager-777c6bd7d5-mrvg2\" (UID: \"a127e29d-b809-4399-a221-c740bfb18d64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"
Sep 14 02:58:33.073195 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:33.073131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a127e29d-b809-4399-a221-c740bfb18d64-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mrvg2\" (UID: \"a127e29d-b809-4399-a221-c740bfb18d64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"
Sep 14 02:58:33.174335 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:33.174275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a127e29d-b809-4399-a221-c740bfb18d64-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mrvg2\" (UID: \"a127e29d-b809-4399-a221-c740bfb18d64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"
Sep 14 02:58:33.174437 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:33.174338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr85x\" (UniqueName: \"kubernetes.io/projected/a127e29d-b809-4399-a221-c740bfb18d64-kube-api-access-fr85x\") pod \"llmisvc-controller-manager-777c6bd7d5-mrvg2\" (UID: \"a127e29d-b809-4399-a221-c740bfb18d64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"
Sep 14 02:58:33.176759 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:33.176737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a127e29d-b809-4399-a221-c740bfb18d64-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mrvg2\" (UID: \"a127e29d-b809-4399-a221-c740bfb18d64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"
Sep 14 02:58:33.208823 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:33.208799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr85x\" (UniqueName: \"kubernetes.io/projected/a127e29d-b809-4399-a221-c740bfb18d64-kube-api-access-fr85x\") pod \"llmisvc-controller-manager-777c6bd7d5-mrvg2\" (UID: \"a127e29d-b809-4399-a221-c740bfb18d64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"
Sep 14 02:58:33.210632 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:33.210613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"
Sep 14 02:58:33.505474 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:33.505411    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"]
Sep 14 02:58:33.507799 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:58:33.507774    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda127e29d_b809_4399_a221_c740bfb18d64.slice/crio-0a3d6a1b0614249fb6972e6a9b4dc1c470d580e61c55fec828a055c9788a996a WatchSource:0}: Error finding container 0a3d6a1b0614249fb6972e6a9b4dc1c470d580e61c55fec828a055c9788a996a: Status 404 returned error can't find the container with id 0a3d6a1b0614249fb6972e6a9b4dc1c470d580e61c55fec828a055c9788a996a
Sep 14 02:58:34.513722 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:34.513686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2" event={"ID":"a127e29d-b809-4399-a221-c740bfb18d64","Type":"ContainerStarted","Data":"0a3d6a1b0614249fb6972e6a9b4dc1c470d580e61c55fec828a055c9788a996a"}
Sep 14 02:58:36.476581 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:36.476545    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mmsrq"
Sep 14 02:58:36.526582 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:36.526458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2" event={"ID":"a127e29d-b809-4399-a221-c740bfb18d64","Type":"ContainerStarted","Data":"1cad7fbb78001100fe05273c5feb909cdffbb4bf73bee2a5a799f049e3c8a48c"}
Sep 14 02:58:36.526582 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:36.526543    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"
Sep 14 02:58:36.561311 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:36.561263    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2" podStartSLOduration=2.263118156 podStartE2EDuration="4.561245779s" totalImagesPullingTime="2.298127623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:58:36.560369039 +0000 UTC m=+383.286327091" watchObservedRunningTime="2026-09-14 02:58:36.561245779 +0000 UTC m=+383.287203831"
Sep 14 02:58:39.536019 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:39.535980    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:39.536483 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:39.536064    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:39.540728 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:39.540706    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:40.548358 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:40.548331    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-666bf7cc8f-v7qng"
Sep 14 02:58:40.618513 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:58:40.618478    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c576996f7-j7d57"]
Sep 14 02:59:05.639138 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.639099    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c576996f7-j7d57" podUID="dccb7399-dba3-4e76-b861-5a109fff882c" containerName="console" containerID="cri-o://fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636" gracePeriod=15
Sep 14 02:59:05.881675 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.881654    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:59:05.914682 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.914618    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-serving-cert\") pod \"dccb7399-dba3-4e76-b861-5a109fff882c\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") "
Sep 14 02:59:05.914800 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.914681    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-trusted-ca-bundle\") pod \"dccb7399-dba3-4e76-b861-5a109fff882c\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") "
Sep 14 02:59:05.914800 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.914757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-oauth-serving-cert\") pod \"dccb7399-dba3-4e76-b861-5a109fff882c\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") "
Sep 14 02:59:05.914918 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.914801    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-console-config\") pod \"dccb7399-dba3-4e76-b861-5a109fff882c\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") "
Sep 14 02:59:05.914918 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.914837    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh8q4\" (UniqueName: \"kubernetes.io/projected/dccb7399-dba3-4e76-b861-5a109fff882c-kube-api-access-xh8q4\") pod \"dccb7399-dba3-4e76-b861-5a109fff882c\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") "
Sep 14 02:59:05.914918 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.914867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-oauth-config\") pod \"dccb7399-dba3-4e76-b861-5a109fff882c\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") "
Sep 14 02:59:05.914918 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.914894    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-service-ca\") pod \"dccb7399-dba3-4e76-b861-5a109fff882c\" (UID: \"dccb7399-dba3-4e76-b861-5a109fff882c\") "
Sep 14 02:59:05.915187 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.915159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dccb7399-dba3-4e76-b861-5a109fff882c" (UID: "dccb7399-dba3-4e76-b861-5a109fff882c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:59:05.915262 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.915212    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-console-config" (OuterVolumeSpecName: "console-config") pod "dccb7399-dba3-4e76-b861-5a109fff882c" (UID: "dccb7399-dba3-4e76-b861-5a109fff882c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:59:05.915521 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.915480    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-service-ca" (OuterVolumeSpecName: "service-ca") pod "dccb7399-dba3-4e76-b861-5a109fff882c" (UID: "dccb7399-dba3-4e76-b861-5a109fff882c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:59:05.915521 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.915491    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dccb7399-dba3-4e76-b861-5a109fff882c" (UID: "dccb7399-dba3-4e76-b861-5a109fff882c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:59:05.917104 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.917073    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dccb7399-dba3-4e76-b861-5a109fff882c" (UID: "dccb7399-dba3-4e76-b861-5a109fff882c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:59:05.917104 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.917086    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccb7399-dba3-4e76-b861-5a109fff882c-kube-api-access-xh8q4" (OuterVolumeSpecName: "kube-api-access-xh8q4") pod "dccb7399-dba3-4e76-b861-5a109fff882c" (UID: "dccb7399-dba3-4e76-b861-5a109fff882c"). InnerVolumeSpecName "kube-api-access-xh8q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:59:05.917245 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:05.917201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dccb7399-dba3-4e76-b861-5a109fff882c" (UID: "dccb7399-dba3-4e76-b861-5a109fff882c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:59:06.016045 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.016014    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-trusted-ca-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:06.016180 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.016057    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-oauth-serving-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:06.016180 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.016068    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-console-config\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:06.016180 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.016087    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh8q4\" (UniqueName: \"kubernetes.io/projected/dccb7399-dba3-4e76-b861-5a109fff882c-kube-api-access-xh8q4\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:06.016180 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.016096    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-oauth-config\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:06.016180 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.016104    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dccb7399-dba3-4e76-b861-5a109fff882c-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:06.016180 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.016112    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dccb7399-dba3-4e76-b861-5a109fff882c-console-serving-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:06.662022 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.661979    2580 generic.go:356] "Generic (PLEG): container finished" podID="dccb7399-dba3-4e76-b861-5a109fff882c" containerID="fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636" exitCode=0
Sep 14 02:59:06.662450 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.662047    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c576996f7-j7d57"
Sep 14 02:59:06.662450 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.662063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c576996f7-j7d57" event={"ID":"dccb7399-dba3-4e76-b861-5a109fff882c","Type":"ContainerDied","Data":"fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636"}
Sep 14 02:59:06.662450 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.662147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c576996f7-j7d57" event={"ID":"dccb7399-dba3-4e76-b861-5a109fff882c","Type":"ContainerDied","Data":"cd04cdce6baf205cfdfce0cb34b2aa9b31fcbb28732d0e45c0ea3d7420e1b262"}
Sep 14 02:59:06.662450 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.662173    2580 scope.go:117] "RemoveContainer" containerID="fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636"
Sep 14 02:59:06.671494 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.671477    2580 scope.go:117] "RemoveContainer" containerID="fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636"
Sep 14 02:59:06.671734 ip-10-0-138-23 kubenswrapper[2580]: E0914 02:59:06.671717    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636\": container with ID starting with fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636 not found: ID does not exist" containerID="fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636"
Sep 14 02:59:06.671779 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.671741    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636"} err="failed to get container status \"fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636\": rpc error: code = NotFound desc = could not find container \"fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636\": container with ID starting with fa33eb6dfd5d78ef2212d2a9dc3273b2757ef714f5882f8494f5ccc727d25636 not found: ID does not exist"
Sep 14 02:59:06.683438 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.683419    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c576996f7-j7d57"]
Sep 14 02:59:06.686593 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:06.686574    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c576996f7-j7d57"]
Sep 14 02:59:07.533702 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:07.533668    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mrvg2"
Sep 14 02:59:07.755705 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:07.755673    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dccb7399-dba3-4e76-b861-5a109fff882c" path="/var/lib/kubelet/pods/dccb7399-dba3-4e76-b861-5a109fff882c/volumes"
Sep 14 02:59:26.850390 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.850355    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"]
Sep 14 02:59:26.850814 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.850633    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dccb7399-dba3-4e76-b861-5a109fff882c" containerName="console"
Sep 14 02:59:26.850814 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.850644    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dccb7399-dba3-4e76-b861-5a109fff882c" containerName="console"
Sep 14 02:59:26.850814 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.850706    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dccb7399-dba3-4e76-b861-5a109fff882c" containerName="console"
Sep 14 02:59:26.855107 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.855089    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:26.857691 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.857669    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 02:59:26.857799 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.857707    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 02:59:26.858322 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.858306    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c2rkr\""
Sep 14 02:59:26.862981 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.862960    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"]
Sep 14 02:59:26.952282 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.952256    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"]
Sep 14 02:59:26.955323 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.955308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:26.965846 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.965824    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"]
Sep 14 02:59:26.978467 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.978448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:26.978552 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.978479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hd6g\" (UniqueName: \"kubernetes.io/projected/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-kube-api-access-7hd6g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:26.978552 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:26.978524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:27.045412 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.045391    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"]
Sep 14 02:59:27.048602 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.048587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:27.058345 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.058321    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"]
Sep 14 02:59:27.079855 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.079829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:27.079951 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.079868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hd6g\" (UniqueName: \"kubernetes.io/projected/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-kube-api-access-7hd6g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:27.079951 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.079889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:27.080053 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.080011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4fml\" (UniqueName: \"kubernetes.io/projected/571909bf-73f4-45c8-9077-38b1de6fcd50-kube-api-access-k4fml\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:27.080133 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.080115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:27.080171 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.080145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:27.080275 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.080254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:27.080376 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.080360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:27.088419 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.088398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hd6g\" (UniqueName: \"kubernetes.io/projected/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-kube-api-access-7hd6g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:27.146557 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.146513    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"]
Sep 14 02:59:27.149652 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.149638    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:27.160158 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.160141    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"]
Sep 14 02:59:27.165371 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.165350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:27.181266 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.181239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:27.181351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.181289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:27.181351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.181343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:27.181418 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.181401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:27.181456 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.181428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4fml\" (UniqueName: \"kubernetes.io/projected/571909bf-73f4-45c8-9077-38b1de6fcd50-kube-api-access-k4fml\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:27.181511 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.181456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh68s\" (UniqueName: \"kubernetes.io/projected/7f77a51c-db0c-4589-a841-3ee5370c35a7-kube-api-access-wh68s\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:27.181705 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.181684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:27.181763 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.181702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:27.195213 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.195192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4fml\" (UniqueName: \"kubernetes.io/projected/571909bf-73f4-45c8-9077-38b1de6fcd50-kube-api-access-k4fml\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:27.263515 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.263485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:27.282608 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.282579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:27.282714 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.282648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:27.282714 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.282682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:27.282789 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.282722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hst2q\" (UniqueName: \"kubernetes.io/projected/477e9c1b-f45f-4d18-84ac-fc2b075c5110-kube-api-access-hst2q\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:27.282789 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.282762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:27.282880 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.282858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh68s\" (UniqueName: \"kubernetes.io/projected/7f77a51c-db0c-4589-a841-3ee5370c35a7-kube-api-access-wh68s\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:27.282962 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.282946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:27.283106 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.283088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:27.309849 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.309814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh68s\" (UniqueName: \"kubernetes.io/projected/7f77a51c-db0c-4589-a841-3ee5370c35a7-kube-api-access-wh68s\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:27.357788 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.357762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:27.383882 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.383849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:27.384006 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.383918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:27.384006 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.383959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hst2q\" (UniqueName: \"kubernetes.io/projected/477e9c1b-f45f-4d18-84ac-fc2b075c5110-kube-api-access-hst2q\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:27.384296 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.384271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:27.384296 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.384287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:27.393469 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.393445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hst2q\" (UniqueName: \"kubernetes.io/projected/477e9c1b-f45f-4d18-84ac-fc2b075c5110-kube-api-access-hst2q\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:27.435126 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.435099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"]
Sep 14 02:59:27.437240 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:59:27.437218    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cdeba5_daaf_4ec8_b209_cb41c777a20c.slice/crio-e3d6e104e1e8aecdb1827f8bfaaf139f7e9e0951c91451ab56ff3154c43750e9 WatchSource:0}: Error finding container e3d6e104e1e8aecdb1827f8bfaaf139f7e9e0951c91451ab56ff3154c43750e9: Status 404 returned error can't find the container with id e3d6e104e1e8aecdb1827f8bfaaf139f7e9e0951c91451ab56ff3154c43750e9
Sep 14 02:59:27.458504 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.458487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:27.536615 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.536585    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"]
Sep 14 02:59:27.539274 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:59:27.539249    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571909bf_73f4_45c8_9077_38b1de6fcd50.slice/crio-2060915a25d4c9c79186d5fd09a88e1c8c603fe8fa86f79f8e39a372dff48f0f WatchSource:0}: Error finding container 2060915a25d4c9c79186d5fd09a88e1c8c603fe8fa86f79f8e39a372dff48f0f: Status 404 returned error can't find the container with id 2060915a25d4c9c79186d5fd09a88e1c8c603fe8fa86f79f8e39a372dff48f0f
Sep 14 02:59:27.630700 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.630677    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"]
Sep 14 02:59:27.632293 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:59:27.632270    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f77a51c_db0c_4589_a841_3ee5370c35a7.slice/crio-028ee07acf64e4e4cc52b8a996b86c9c52d69c6561804a810ecfcb5c218d000f WatchSource:0}: Error finding container 028ee07acf64e4e4cc52b8a996b86c9c52d69c6561804a810ecfcb5c218d000f: Status 404 returned error can't find the container with id 028ee07acf64e4e4cc52b8a996b86c9c52d69c6561804a810ecfcb5c218d000f
Sep 14 02:59:27.731295 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.731180    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"]
Sep 14 02:59:27.758574 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.758547    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f77a51c-db0c-4589-a841-3ee5370c35a7" containerID="2d2afe84d3cbc5f4292e817d88eace9e28b65f3acca216d3275e72ede2376540" exitCode=0
Sep 14 02:59:27.758713 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.758625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd" event={"ID":"7f77a51c-db0c-4589-a841-3ee5370c35a7","Type":"ContainerDied","Data":"2d2afe84d3cbc5f4292e817d88eace9e28b65f3acca216d3275e72ede2376540"}
Sep 14 02:59:27.758713 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.758665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd" event={"ID":"7f77a51c-db0c-4589-a841-3ee5370c35a7","Type":"ContainerStarted","Data":"028ee07acf64e4e4cc52b8a996b86c9c52d69c6561804a810ecfcb5c218d000f"}
Sep 14 02:59:27.760262 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.760237    2580 generic.go:356] "Generic (PLEG): container finished" podID="571909bf-73f4-45c8-9077-38b1de6fcd50" containerID="8ae39e3cbc3aafdad35d50b7add7874161b7c2d90ef8600e5e2da15cbadc7e8c" exitCode=0
Sep 14 02:59:27.760368 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.760328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r" event={"ID":"571909bf-73f4-45c8-9077-38b1de6fcd50","Type":"ContainerDied","Data":"8ae39e3cbc3aafdad35d50b7add7874161b7c2d90ef8600e5e2da15cbadc7e8c"}
Sep 14 02:59:27.760427 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.760374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r" event={"ID":"571909bf-73f4-45c8-9077-38b1de6fcd50","Type":"ContainerStarted","Data":"2060915a25d4c9c79186d5fd09a88e1c8c603fe8fa86f79f8e39a372dff48f0f"}
Sep 14 02:59:27.761960 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.761938    2580 generic.go:356] "Generic (PLEG): container finished" podID="c3cdeba5-daaf-4ec8-b209-cb41c777a20c" containerID="6b18d58becf03b3b0499727f83cb2b2f693b33cf53776a9c824958ba93cae934" exitCode=0
Sep 14 02:59:27.762081 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.761988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn" event={"ID":"c3cdeba5-daaf-4ec8-b209-cb41c777a20c","Type":"ContainerDied","Data":"6b18d58becf03b3b0499727f83cb2b2f693b33cf53776a9c824958ba93cae934"}
Sep 14 02:59:27.762081 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:27.762055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn" event={"ID":"c3cdeba5-daaf-4ec8-b209-cb41c777a20c","Type":"ContainerStarted","Data":"e3d6e104e1e8aecdb1827f8bfaaf139f7e9e0951c91451ab56ff3154c43750e9"}
Sep 14 02:59:27.778196 ip-10-0-138-23 kubenswrapper[2580]: W0914 02:59:27.778173    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477e9c1b_f45f_4d18_84ac_fc2b075c5110.slice/crio-e7f58330c48160b32898ffba8520a84002ab31920aa37612fc000c4fd355f242 WatchSource:0}: Error finding container e7f58330c48160b32898ffba8520a84002ab31920aa37612fc000c4fd355f242: Status 404 returned error can't find the container with id e7f58330c48160b32898ffba8520a84002ab31920aa37612fc000c4fd355f242
Sep 14 02:59:28.768973 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:28.768941    2580 generic.go:356] "Generic (PLEG): container finished" podID="c3cdeba5-daaf-4ec8-b209-cb41c777a20c" containerID="c9bc6789f9d358d95e382c32416dd96aedd454de01839a1f58971ff01ff30ae3" exitCode=0
Sep 14 02:59:28.769281 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:28.769077    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn" event={"ID":"c3cdeba5-daaf-4ec8-b209-cb41c777a20c","Type":"ContainerDied","Data":"c9bc6789f9d358d95e382c32416dd96aedd454de01839a1f58971ff01ff30ae3"}
Sep 14 02:59:28.770915 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:28.770888    2580 generic.go:356] "Generic (PLEG): container finished" podID="477e9c1b-f45f-4d18-84ac-fc2b075c5110" containerID="4f7c58cd7fff602a4622dec6d6f8bfbc813d32057cf83385ca163eba3ae011d8" exitCode=0
Sep 14 02:59:28.770996 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:28.770927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7" event={"ID":"477e9c1b-f45f-4d18-84ac-fc2b075c5110","Type":"ContainerDied","Data":"4f7c58cd7fff602a4622dec6d6f8bfbc813d32057cf83385ca163eba3ae011d8"}
Sep 14 02:59:28.770996 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:28.770964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7" event={"ID":"477e9c1b-f45f-4d18-84ac-fc2b075c5110","Type":"ContainerStarted","Data":"e7f58330c48160b32898ffba8520a84002ab31920aa37612fc000c4fd355f242"}
Sep 14 02:59:29.777505 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:29.777479    2580 generic.go:356] "Generic (PLEG): container finished" podID="571909bf-73f4-45c8-9077-38b1de6fcd50" containerID="fa6eb4c690ff7e57e14c378c5966f64dc02a9430f58fb3c9c73fd317518d4be6" exitCode=0
Sep 14 02:59:29.777969 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:29.777546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r" event={"ID":"571909bf-73f4-45c8-9077-38b1de6fcd50","Type":"ContainerDied","Data":"fa6eb4c690ff7e57e14c378c5966f64dc02a9430f58fb3c9c73fd317518d4be6"}
Sep 14 02:59:29.779476 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:29.779451    2580 generic.go:356] "Generic (PLEG): container finished" podID="477e9c1b-f45f-4d18-84ac-fc2b075c5110" containerID="690c853a926c14ddc947b4dbb28e9bae1e3f7e585f38a39eaa3d90add0dc57cd" exitCode=0
Sep 14 02:59:29.779565 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:29.779533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7" event={"ID":"477e9c1b-f45f-4d18-84ac-fc2b075c5110","Type":"ContainerDied","Data":"690c853a926c14ddc947b4dbb28e9bae1e3f7e585f38a39eaa3d90add0dc57cd"}
Sep 14 02:59:29.781710 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:29.781689    2580 generic.go:356] "Generic (PLEG): container finished" podID="c3cdeba5-daaf-4ec8-b209-cb41c777a20c" containerID="f13b3ae68d487bde907ae7239e62d1dd56e44f70f5e8100f35974aed07d37065" exitCode=0
Sep 14 02:59:29.781789 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:29.781725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn" event={"ID":"c3cdeba5-daaf-4ec8-b209-cb41c777a20c","Type":"ContainerDied","Data":"f13b3ae68d487bde907ae7239e62d1dd56e44f70f5e8100f35974aed07d37065"}
Sep 14 02:59:30.789071 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:30.789007    2580 generic.go:356] "Generic (PLEG): container finished" podID="571909bf-73f4-45c8-9077-38b1de6fcd50" containerID="1803e2bf4ed01a875c0173760e40a5b3cef734054ebe0e5993ed8e380825b253" exitCode=0
Sep 14 02:59:30.789473 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:30.789094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r" event={"ID":"571909bf-73f4-45c8-9077-38b1de6fcd50","Type":"ContainerDied","Data":"1803e2bf4ed01a875c0173760e40a5b3cef734054ebe0e5993ed8e380825b253"}
Sep 14 02:59:30.791143 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:30.791121    2580 generic.go:356] "Generic (PLEG): container finished" podID="477e9c1b-f45f-4d18-84ac-fc2b075c5110" containerID="36559b94aaf4219ae4be6ef8f1373a0073e7a211a2c5cfdb1ea6760c27a13c0f" exitCode=0
Sep 14 02:59:30.791241 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:30.791211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7" event={"ID":"477e9c1b-f45f-4d18-84ac-fc2b075c5110","Type":"ContainerDied","Data":"36559b94aaf4219ae4be6ef8f1373a0073e7a211a2c5cfdb1ea6760c27a13c0f"}
Sep 14 02:59:30.916547 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:30.916523    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:31.013413 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.013385    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hd6g\" (UniqueName: \"kubernetes.io/projected/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-kube-api-access-7hd6g\") pod \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") "
Sep 14 02:59:31.013526 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.013465    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-util\") pod \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") "
Sep 14 02:59:31.013526 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.013510    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-bundle\") pod \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\" (UID: \"c3cdeba5-daaf-4ec8-b209-cb41c777a20c\") "
Sep 14 02:59:31.013949 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.013923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-bundle" (OuterVolumeSpecName: "bundle") pod "c3cdeba5-daaf-4ec8-b209-cb41c777a20c" (UID: "c3cdeba5-daaf-4ec8-b209-cb41c777a20c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:59:31.015424 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.015404    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-kube-api-access-7hd6g" (OuterVolumeSpecName: "kube-api-access-7hd6g") pod "c3cdeba5-daaf-4ec8-b209-cb41c777a20c" (UID: "c3cdeba5-daaf-4ec8-b209-cb41c777a20c"). InnerVolumeSpecName "kube-api-access-7hd6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:59:31.019221 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.019201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-util" (OuterVolumeSpecName: "util") pod "c3cdeba5-daaf-4ec8-b209-cb41c777a20c" (UID: "c3cdeba5-daaf-4ec8-b209-cb41c777a20c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:59:31.114093 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.114022    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-util\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:31.114093 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.114063    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:31.114093 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.114072    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hd6g\" (UniqueName: \"kubernetes.io/projected/c3cdeba5-daaf-4ec8-b209-cb41c777a20c-kube-api-access-7hd6g\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:31.799296 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.799271    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn"
Sep 14 02:59:31.799670 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.799305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h44kn" event={"ID":"c3cdeba5-daaf-4ec8-b209-cb41c777a20c","Type":"ContainerDied","Data":"e3d6e104e1e8aecdb1827f8bfaaf139f7e9e0951c91451ab56ff3154c43750e9"}
Sep 14 02:59:31.799670 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.799353    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3d6e104e1e8aecdb1827f8bfaaf139f7e9e0951c91451ab56ff3154c43750e9"
Sep 14 02:59:31.953109 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.953082    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:31.958813 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:31.958793    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:32.123558 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.123485    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-bundle\") pod \"571909bf-73f4-45c8-9077-38b1de6fcd50\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") "
Sep 14 02:59:32.123558 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.123537    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-bundle\") pod \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") "
Sep 14 02:59:32.123763 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.123582    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-util\") pod \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") "
Sep 14 02:59:32.123763 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.123615    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4fml\" (UniqueName: \"kubernetes.io/projected/571909bf-73f4-45c8-9077-38b1de6fcd50-kube-api-access-k4fml\") pod \"571909bf-73f4-45c8-9077-38b1de6fcd50\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") "
Sep 14 02:59:32.123763 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.123647    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-util\") pod \"571909bf-73f4-45c8-9077-38b1de6fcd50\" (UID: \"571909bf-73f4-45c8-9077-38b1de6fcd50\") "
Sep 14 02:59:32.123763 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.123693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hst2q\" (UniqueName: \"kubernetes.io/projected/477e9c1b-f45f-4d18-84ac-fc2b075c5110-kube-api-access-hst2q\") pod \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\" (UID: \"477e9c1b-f45f-4d18-84ac-fc2b075c5110\") "
Sep 14 02:59:32.124250 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.124221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-bundle" (OuterVolumeSpecName: "bundle") pod "477e9c1b-f45f-4d18-84ac-fc2b075c5110" (UID: "477e9c1b-f45f-4d18-84ac-fc2b075c5110"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:59:32.124418 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.124394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-bundle" (OuterVolumeSpecName: "bundle") pod "571909bf-73f4-45c8-9077-38b1de6fcd50" (UID: "571909bf-73f4-45c8-9077-38b1de6fcd50"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:59:32.125728 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.125697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477e9c1b-f45f-4d18-84ac-fc2b075c5110-kube-api-access-hst2q" (OuterVolumeSpecName: "kube-api-access-hst2q") pod "477e9c1b-f45f-4d18-84ac-fc2b075c5110" (UID: "477e9c1b-f45f-4d18-84ac-fc2b075c5110"). InnerVolumeSpecName "kube-api-access-hst2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:59:32.125952 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.125925    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571909bf-73f4-45c8-9077-38b1de6fcd50-kube-api-access-k4fml" (OuterVolumeSpecName: "kube-api-access-k4fml") pod "571909bf-73f4-45c8-9077-38b1de6fcd50" (UID: "571909bf-73f4-45c8-9077-38b1de6fcd50"). InnerVolumeSpecName "kube-api-access-k4fml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:59:32.129509 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.129474    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-util" (OuterVolumeSpecName: "util") pod "477e9c1b-f45f-4d18-84ac-fc2b075c5110" (UID: "477e9c1b-f45f-4d18-84ac-fc2b075c5110"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:59:32.129694 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.129670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-util" (OuterVolumeSpecName: "util") pod "571909bf-73f4-45c8-9077-38b1de6fcd50" (UID: "571909bf-73f4-45c8-9077-38b1de6fcd50"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:59:32.225236 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.225211    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:32.225236 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.225233    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:32.225236 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.225241    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/477e9c1b-f45f-4d18-84ac-fc2b075c5110-util\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:32.225393 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.225250    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4fml\" (UniqueName: \"kubernetes.io/projected/571909bf-73f4-45c8-9077-38b1de6fcd50-kube-api-access-k4fml\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:32.225393 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.225260    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/571909bf-73f4-45c8-9077-38b1de6fcd50-util\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:32.225393 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.225268    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hst2q\" (UniqueName: \"kubernetes.io/projected/477e9c1b-f45f-4d18-84ac-fc2b075c5110-kube-api-access-hst2q\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:32.805802 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.805769    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r"
Sep 14 02:59:32.806232 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.805797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afzc4r" event={"ID":"571909bf-73f4-45c8-9077-38b1de6fcd50","Type":"ContainerDied","Data":"2060915a25d4c9c79186d5fd09a88e1c8c603fe8fa86f79f8e39a372dff48f0f"}
Sep 14 02:59:32.806232 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.805840    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2060915a25d4c9c79186d5fd09a88e1c8c603fe8fa86f79f8e39a372dff48f0f"
Sep 14 02:59:32.807693 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.807668    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7" event={"ID":"477e9c1b-f45f-4d18-84ac-fc2b075c5110","Type":"ContainerDied","Data":"e7f58330c48160b32898ffba8520a84002ab31920aa37612fc000c4fd355f242"}
Sep 14 02:59:32.807693 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.807683    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chhzb7"
Sep 14 02:59:32.807841 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:32.807702    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7f58330c48160b32898ffba8520a84002ab31920aa37612fc000c4fd355f242"
Sep 14 02:59:49.890855 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:49.890780    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f77a51c-db0c-4589-a841-3ee5370c35a7" containerID="6deb033578bfdcedc6d2f1f2bedd75d5fa5df0cbe74dd2c120d3fad3cda50a1b" exitCode=0
Sep 14 02:59:49.890855 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:49.890810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd" event={"ID":"7f77a51c-db0c-4589-a841-3ee5370c35a7","Type":"ContainerDied","Data":"6deb033578bfdcedc6d2f1f2bedd75d5fa5df0cbe74dd2c120d3fad3cda50a1b"}
Sep 14 02:59:50.897764 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:50.897727    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f77a51c-db0c-4589-a841-3ee5370c35a7" containerID="e2d9c5790367c271097321a1c84658a11b4cbec87b5534d660d6ab750ca7a8ac" exitCode=0
Sep 14 02:59:50.898203 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:50.897816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd" event={"ID":"7f77a51c-db0c-4589-a841-3ee5370c35a7","Type":"ContainerDied","Data":"e2d9c5790367c271097321a1c84658a11b4cbec87b5534d660d6ab750ca7a8ac"}
Sep 14 02:59:52.026935 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.026908    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:52.073824 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.073797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh68s\" (UniqueName: \"kubernetes.io/projected/7f77a51c-db0c-4589-a841-3ee5370c35a7-kube-api-access-wh68s\") pod \"7f77a51c-db0c-4589-a841-3ee5370c35a7\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") "
Sep 14 02:59:52.073961 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.073832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-bundle\") pod \"7f77a51c-db0c-4589-a841-3ee5370c35a7\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") "
Sep 14 02:59:52.073961 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.073950    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-util\") pod \"7f77a51c-db0c-4589-a841-3ee5370c35a7\" (UID: \"7f77a51c-db0c-4589-a841-3ee5370c35a7\") "
Sep 14 02:59:52.074362 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.074337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-bundle" (OuterVolumeSpecName: "bundle") pod "7f77a51c-db0c-4589-a841-3ee5370c35a7" (UID: "7f77a51c-db0c-4589-a841-3ee5370c35a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:59:52.075768 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.075746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f77a51c-db0c-4589-a841-3ee5370c35a7-kube-api-access-wh68s" (OuterVolumeSpecName: "kube-api-access-wh68s") pod "7f77a51c-db0c-4589-a841-3ee5370c35a7" (UID: "7f77a51c-db0c-4589-a841-3ee5370c35a7"). InnerVolumeSpecName "kube-api-access-wh68s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:59:52.078443 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.078422    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-util" (OuterVolumeSpecName: "util") pod "7f77a51c-db0c-4589-a841-3ee5370c35a7" (UID: "7f77a51c-db0c-4589-a841-3ee5370c35a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:59:52.174572 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.174518    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh68s\" (UniqueName: \"kubernetes.io/projected/7f77a51c-db0c-4589-a841-3ee5370c35a7-kube-api-access-wh68s\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:52.174572 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.174539    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:52.174572 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.174548    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f77a51c-db0c-4589-a841-3ee5370c35a7-util\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 02:59:52.909216 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.909193    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd"
Sep 14 02:59:52.909351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.909219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jtfrd" event={"ID":"7f77a51c-db0c-4589-a841-3ee5370c35a7","Type":"ContainerDied","Data":"028ee07acf64e4e4cc52b8a996b86c9c52d69c6561804a810ecfcb5c218d000f"}
Sep 14 02:59:52.909351 ip-10-0-138-23 kubenswrapper[2580]: I0914 02:59:52.909257    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="028ee07acf64e4e4cc52b8a996b86c9c52d69c6561804a810ecfcb5c218d000f"
Sep 14 03:00:01.748374 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748338    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"]
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748625    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f77a51c-db0c-4589-a841-3ee5370c35a7" containerName="pull"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748635    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f77a51c-db0c-4589-a841-3ee5370c35a7" containerName="pull"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748646    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="571909bf-73f4-45c8-9077-38b1de6fcd50" containerName="util"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748651    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="571909bf-73f4-45c8-9077-38b1de6fcd50" containerName="util"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748658    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3cdeba5-daaf-4ec8-b209-cb41c777a20c" containerName="util"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748664    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3cdeba5-daaf-4ec8-b209-cb41c777a20c" containerName="util"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748670    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3cdeba5-daaf-4ec8-b209-cb41c777a20c" containerName="pull"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748675    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3cdeba5-daaf-4ec8-b209-cb41c777a20c" containerName="pull"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748683    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f77a51c-db0c-4589-a841-3ee5370c35a7" containerName="util"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748689    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f77a51c-db0c-4589-a841-3ee5370c35a7" containerName="util"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748696    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="571909bf-73f4-45c8-9077-38b1de6fcd50" containerName="extract"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748701    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="571909bf-73f4-45c8-9077-38b1de6fcd50" containerName="extract"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748707    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3cdeba5-daaf-4ec8-b209-cb41c777a20c" containerName="extract"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748712    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3cdeba5-daaf-4ec8-b209-cb41c777a20c" containerName="extract"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748717    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="477e9c1b-f45f-4d18-84ac-fc2b075c5110" containerName="extract"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748722    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="477e9c1b-f45f-4d18-84ac-fc2b075c5110" containerName="extract"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748729    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f77a51c-db0c-4589-a841-3ee5370c35a7" containerName="extract"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748734    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f77a51c-db0c-4589-a841-3ee5370c35a7" containerName="extract"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748742    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="477e9c1b-f45f-4d18-84ac-fc2b075c5110" containerName="util"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748747    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="477e9c1b-f45f-4d18-84ac-fc2b075c5110" containerName="util"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748755    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="571909bf-73f4-45c8-9077-38b1de6fcd50" containerName="pull"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748759    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="571909bf-73f4-45c8-9077-38b1de6fcd50" containerName="pull"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748765    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="477e9c1b-f45f-4d18-84ac-fc2b075c5110" containerName="pull"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748769    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="477e9c1b-f45f-4d18-84ac-fc2b075c5110" containerName="pull"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748814    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3cdeba5-daaf-4ec8-b209-cb41c777a20c" containerName="extract"
Sep 14 03:00:01.748809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748823    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="571909bf-73f4-45c8-9077-38b1de6fcd50" containerName="extract"
Sep 14 03:00:01.749578 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748829    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="477e9c1b-f45f-4d18-84ac-fc2b075c5110" containerName="extract"
Sep 14 03:00:01.749578 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.748837    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f77a51c-db0c-4589-a841-3ee5370c35a7" containerName="extract"
Sep 14 03:00:01.753065 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.753042    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:01.756868 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.756847    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 14 03:00:01.757331 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.757310    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pq827\""
Sep 14 03:00:01.757634 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.757621    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 14 03:00:01.777068 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.777042    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"]
Sep 14 03:00:01.849617 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.849586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lq9m\" (UniqueName: \"kubernetes.io/projected/d813c8d3-955a-4de6-a504-de917b77e8d7-kube-api-access-8lq9m\") pod \"limitador-operator-controller-manager-74495c69f8-tvcrp\" (UID: \"d813c8d3-955a-4de6-a504-de917b77e8d7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:01.950817 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.950790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lq9m\" (UniqueName: \"kubernetes.io/projected/d813c8d3-955a-4de6-a504-de917b77e8d7-kube-api-access-8lq9m\") pod \"limitador-operator-controller-manager-74495c69f8-tvcrp\" (UID: \"d813c8d3-955a-4de6-a504-de917b77e8d7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:01.964205 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:01.964181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lq9m\" (UniqueName: \"kubernetes.io/projected/d813c8d3-955a-4de6-a504-de917b77e8d7-kube-api-access-8lq9m\") pod \"limitador-operator-controller-manager-74495c69f8-tvcrp\" (UID: \"d813c8d3-955a-4de6-a504-de917b77e8d7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:02.063496 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:02.063460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:02.345387 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:02.345361    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"]
Sep 14 03:00:02.347203 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:00:02.347171    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd813c8d3_955a_4de6_a504_de917b77e8d7.slice/crio-1e9f4298e5f1d47206a2d33861976b6cc111c140196477ee627a56b5f2df3a19 WatchSource:0}: Error finding container 1e9f4298e5f1d47206a2d33861976b6cc111c140196477ee627a56b5f2df3a19: Status 404 returned error can't find the container with id 1e9f4298e5f1d47206a2d33861976b6cc111c140196477ee627a56b5f2df3a19
Sep 14 03:00:02.963283 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:02.963248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp" event={"ID":"d813c8d3-955a-4de6-a504-de917b77e8d7","Type":"ContainerStarted","Data":"1e9f4298e5f1d47206a2d33861976b6cc111c140196477ee627a56b5f2df3a19"}
Sep 14 03:00:06.988345 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:06.988305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp" event={"ID":"d813c8d3-955a-4de6-a504-de917b77e8d7","Type":"ContainerStarted","Data":"1cbcc05c74422a2f2a38a089b28f36b33f42ad5fb063ac80788f19e6a4a139fa"}
Sep 14 03:00:06.988779 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:06.988389    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:07.015551 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:07.015503    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp" podStartSLOduration=1.980598158 podStartE2EDuration="6.015488632s" totalImagesPullingTime="4.034890474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:00:07.01414128 +0000 UTC m=+473.740099332" watchObservedRunningTime="2026-09-14 03:00:07.015488632 +0000 UTC m=+473.741446685"
Sep 14 03:00:16.601590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.601555    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"]
Sep 14 03:00:16.604736 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.604718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"
Sep 14 03:00:16.612452 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.612429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 14 03:00:16.612552 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.612528    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ztb9t\""
Sep 14 03:00:16.618593 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.618577    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 14 03:00:16.638779 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.638756    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"]
Sep 14 03:00:16.668705 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.668681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ae1db549-83b6-437e-ac63-0e4f4c97127f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bvpp2\" (UID: \"ae1db549-83b6-437e-ac63-0e4f4c97127f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"
Sep 14 03:00:16.668801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.668712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae1db549-83b6-437e-ac63-0e4f4c97127f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bvpp2\" (UID: \"ae1db549-83b6-437e-ac63-0e4f4c97127f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"
Sep 14 03:00:16.668801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.668740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn9mt\" (UniqueName: \"kubernetes.io/projected/ae1db549-83b6-437e-ac63-0e4f4c97127f-kube-api-access-pn9mt\") pod \"kuadrant-console-plugin-fbdd4f445-bvpp2\" (UID: \"ae1db549-83b6-437e-ac63-0e4f4c97127f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"
Sep 14 03:00:16.769211 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.769188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ae1db549-83b6-437e-ac63-0e4f4c97127f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bvpp2\" (UID: \"ae1db549-83b6-437e-ac63-0e4f4c97127f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"
Sep 14 03:00:16.769329 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.769220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae1db549-83b6-437e-ac63-0e4f4c97127f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bvpp2\" (UID: \"ae1db549-83b6-437e-ac63-0e4f4c97127f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"
Sep 14 03:00:16.769329 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.769248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn9mt\" (UniqueName: \"kubernetes.io/projected/ae1db549-83b6-437e-ac63-0e4f4c97127f-kube-api-access-pn9mt\") pod \"kuadrant-console-plugin-fbdd4f445-bvpp2\" (UID: \"ae1db549-83b6-437e-ac63-0e4f4c97127f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"
Sep 14 03:00:16.769843 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.769814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ae1db549-83b6-437e-ac63-0e4f4c97127f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bvpp2\" (UID: \"ae1db549-83b6-437e-ac63-0e4f4c97127f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"
Sep 14 03:00:16.771769 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.771744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae1db549-83b6-437e-ac63-0e4f4c97127f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bvpp2\" (UID: \"ae1db549-83b6-437e-ac63-0e4f4c97127f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"
Sep 14 03:00:16.780154 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.780131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn9mt\" (UniqueName: \"kubernetes.io/projected/ae1db549-83b6-437e-ac63-0e4f4c97127f-kube-api-access-pn9mt\") pod \"kuadrant-console-plugin-fbdd4f445-bvpp2\" (UID: \"ae1db549-83b6-437e-ac63-0e4f4c97127f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"
Sep 14 03:00:16.913058 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:16.912987    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"
Sep 14 03:00:17.191382 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:17.191358    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2"]
Sep 14 03:00:17.193002 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:00:17.192973    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae1db549_83b6_437e_ac63_0e4f4c97127f.slice/crio-c14f262c58e701fd1890b049352e417f035cac9454ce1dac9c466da0529de94b WatchSource:0}: Error finding container c14f262c58e701fd1890b049352e417f035cac9454ce1dac9c466da0529de94b: Status 404 returned error can't find the container with id c14f262c58e701fd1890b049352e417f035cac9454ce1dac9c466da0529de94b
Sep 14 03:00:17.995700 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:17.995673    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:18.054563 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:18.054526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2" event={"ID":"ae1db549-83b6-437e-ac63-0e4f4c97127f","Type":"ContainerStarted","Data":"c14f262c58e701fd1890b049352e417f035cac9454ce1dac9c466da0529de94b"}
Sep 14 03:00:22.078570 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:22.078535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2" event={"ID":"ae1db549-83b6-437e-ac63-0e4f4c97127f","Type":"ContainerStarted","Data":"a7ff57a80fe0a5b7905f43d232b4c56586ab1f1261b45622ce4666750344be34"}
Sep 14 03:00:22.096070 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:22.096008    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bvpp2" podStartSLOduration=1.5421987719999999 podStartE2EDuration="6.095992601s" totalImagesPullingTime="4.553793829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:00:22.094313868 +0000 UTC m=+488.820271921" watchObservedRunningTime="2026-09-14 03:00:22.095992601 +0000 UTC m=+488.821950653"
Sep 14 03:00:27.904922 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:27.904890    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"]
Sep 14 03:00:27.905346 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:27.905110    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp" podUID="d813c8d3-955a-4de6-a504-de917b77e8d7" containerName="manager" containerID="cri-o://1cbcc05c74422a2f2a38a089b28f36b33f42ad5fb063ac80788f19e6a4a139fa" gracePeriod=2
Sep 14 03:00:27.926532 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:27.926507    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"]
Sep 14 03:00:27.934239 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:00:27.934206    2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d813c8d3-955a-4de6-a504-de917b77e8d7\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-tvcrp\": pods \"limitador-operator-controller-manager-74495c69f8-tvcrp\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:28.040632 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.040589    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng"]
Sep 14 03:00:28.040924 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.040909    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d813c8d3-955a-4de6-a504-de917b77e8d7" containerName="manager"
Sep 14 03:00:28.040924 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.040925    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d813c8d3-955a-4de6-a504-de917b77e8d7" containerName="manager"
Sep 14 03:00:28.041097 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.040997    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d813c8d3-955a-4de6-a504-de917b77e8d7" containerName="manager"
Sep 14 03:00:28.044375 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.044348    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng"
Sep 14 03:00:28.047379 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:00:28.047346    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tvcrp\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="d813c8d3-955a-4de6-a504-de917b77e8d7" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:28.102995 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.102966    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng"]
Sep 14 03:00:28.111793 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.111764    2580 generic.go:356] "Generic (PLEG): container finished" podID="d813c8d3-955a-4de6-a504-de917b77e8d7" containerID="1cbcc05c74422a2f2a38a089b28f36b33f42ad5fb063ac80788f19e6a4a139fa" exitCode=0
Sep 14 03:00:28.140766 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.140747    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:28.143429 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:00:28.143405    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tvcrp\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="d813c8d3-955a-4de6-a504-de917b77e8d7" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:28.163868 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.163815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tlzf\" (UniqueName: \"kubernetes.io/projected/1838aa69-6c78-4cde-9036-54771d072762-kube-api-access-4tlzf\") pod \"limitador-operator-controller-manager-74495c69f8-68nng\" (UID: \"1838aa69-6c78-4cde-9036-54771d072762\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng"
Sep 14 03:00:28.264580 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.264557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lq9m\" (UniqueName: \"kubernetes.io/projected/d813c8d3-955a-4de6-a504-de917b77e8d7-kube-api-access-8lq9m\") pod \"d813c8d3-955a-4de6-a504-de917b77e8d7\" (UID: \"d813c8d3-955a-4de6-a504-de917b77e8d7\") "
Sep 14 03:00:28.264746 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.264725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tlzf\" (UniqueName: \"kubernetes.io/projected/1838aa69-6c78-4cde-9036-54771d072762-kube-api-access-4tlzf\") pod \"limitador-operator-controller-manager-74495c69f8-68nng\" (UID: \"1838aa69-6c78-4cde-9036-54771d072762\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng"
Sep 14 03:00:28.266718 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.266696    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d813c8d3-955a-4de6-a504-de917b77e8d7-kube-api-access-8lq9m" (OuterVolumeSpecName: "kube-api-access-8lq9m") pod "d813c8d3-955a-4de6-a504-de917b77e8d7" (UID: "d813c8d3-955a-4de6-a504-de917b77e8d7"). InnerVolumeSpecName "kube-api-access-8lq9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:00:28.274236 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.274216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tlzf\" (UniqueName: \"kubernetes.io/projected/1838aa69-6c78-4cde-9036-54771d072762-kube-api-access-4tlzf\") pod \"limitador-operator-controller-manager-74495c69f8-68nng\" (UID: \"1838aa69-6c78-4cde-9036-54771d072762\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng"
Sep 14 03:00:28.354324 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.354291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng"
Sep 14 03:00:28.365773 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.365745    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lq9m\" (UniqueName: \"kubernetes.io/projected/d813c8d3-955a-4de6-a504-de917b77e8d7-kube-api-access-8lq9m\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:00:28.650052 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.649983    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng"]
Sep 14 03:00:28.652803 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:00:28.652782    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1838aa69_6c78_4cde_9036_54771d072762.slice/crio-c77ce410b127254d5c23c034d4464a0e8b4d5ef6aca13c9e55f8a787873eb29d WatchSource:0}: Error finding container c77ce410b127254d5c23c034d4464a0e8b4d5ef6aca13c9e55f8a787873eb29d: Status 404 returned error can't find the container with id c77ce410b127254d5c23c034d4464a0e8b4d5ef6aca13c9e55f8a787873eb29d
Sep 14 03:00:28.994595 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:28.994507    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp" podUID="d813c8d3-955a-4de6-a504-de917b77e8d7" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 03:00:29.118231 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:29.118200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng" event={"ID":"1838aa69-6c78-4cde-9036-54771d072762","Type":"ContainerStarted","Data":"347c824da0cfc35663f6c1e237f35c9ace5938f44f7ee84507398180449c57bd"}
Sep 14 03:00:29.118389 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:29.118253    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng" event={"ID":"1838aa69-6c78-4cde-9036-54771d072762","Type":"ContainerStarted","Data":"c77ce410b127254d5c23c034d4464a0e8b4d5ef6aca13c9e55f8a787873eb29d"}
Sep 14 03:00:29.118389 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:29.118276    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng"
Sep 14 03:00:29.119610 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:29.119591    2580 scope.go:117] "RemoveContainer" containerID="1cbcc05c74422a2f2a38a089b28f36b33f42ad5fb063ac80788f19e6a4a139fa"
Sep 14 03:00:29.119610 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:29.119602    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:29.174954 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:29.174907    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng" podStartSLOduration=2.17488817 podStartE2EDuration="2.17488817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:00:29.174592622 +0000 UTC m=+495.900550667" watchObservedRunningTime="2026-09-14 03:00:29.17488817 +0000 UTC m=+495.900846224"
Sep 14 03:00:29.176452 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:00:29.176424    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tvcrp\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="d813c8d3-955a-4de6-a504-de917b77e8d7" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:29.179633 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:00:29.179608    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tvcrp\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="d813c8d3-955a-4de6-a504-de917b77e8d7" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tvcrp"
Sep 14 03:00:29.756314 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:29.756272    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d813c8d3-955a-4de6-a504-de917b77e8d7" path="/var/lib/kubelet/pods/d813c8d3-955a-4de6-a504-de917b77e8d7/volumes"
Sep 14 03:00:40.127654 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:00:40.127623    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-68nng"
Sep 14 03:01:03.311639 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.311605    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"]
Sep 14 03:01:03.330810 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.330787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.336268 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.336231    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8tfgh\""
Sep 14 03:01:03.338204 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.338171    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"]
Sep 14 03:01:03.431040 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.431005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.431229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.431064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4f2\" (UniqueName: \"kubernetes.io/projected/65744834-d235-4b52-b655-6b0c87e69c27-kube-api-access-zw4f2\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.431229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.431154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.431229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.431208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.431373 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.431231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65744834-d235-4b52-b655-6b0c87e69c27-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.431373 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.431252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65744834-d235-4b52-b655-6b0c87e69c27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.431373 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.431279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.431373 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.431358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65744834-d235-4b52-b655-6b0c87e69c27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.431579 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.431393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532167 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532167 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65744834-d235-4b52-b655-6b0c87e69c27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4f2\" (UniqueName: \"kubernetes.io/projected/65744834-d235-4b52-b655-6b0c87e69c27-kube-api-access-zw4f2\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532682 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65744834-d235-4b52-b655-6b0c87e69c27-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532682 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65744834-d235-4b52-b655-6b0c87e69c27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532682 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532832 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532832 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.532982 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.532959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65744834-d235-4b52-b655-6b0c87e69c27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.533113 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.533097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.534618 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.534600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65744834-d235-4b52-b655-6b0c87e69c27-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.534844 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.534823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65744834-d235-4b52-b655-6b0c87e69c27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.541509 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.541481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4f2\" (UniqueName: \"kubernetes.io/projected/65744834-d235-4b52-b655-6b0c87e69c27-kube-api-access-zw4f2\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.541948 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.541914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65744834-d235-4b52-b655-6b0c87e69c27-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-bh7f6\" (UID: \"65744834-d235-4b52-b655-6b0c87e69c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.641927 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.641848    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:03.919135 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.919099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"]
Sep 14 03:01:03.925863 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.925827    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:01:03.925955 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.925889    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:01:03.925955 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:03.925915    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:01:04.293089 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:04.293020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6" event={"ID":"65744834-d235-4b52-b655-6b0c87e69c27","Type":"ContainerStarted","Data":"4aa9d674ed9dc332888eac188230b38f0ee9893e94e861ffe4798aea64e9fa43"}
Sep 14 03:01:04.293249 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:04.293112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6" event={"ID":"65744834-d235-4b52-b655-6b0c87e69c27","Type":"ContainerStarted","Data":"fd813e85ddfd8a7b8810790e1585c99d8eeb0428b04fd6d63c65f1158a2d12f0"}
Sep 14 03:01:04.315925 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:04.315882    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6" podStartSLOduration=1.315866929 podStartE2EDuration="1.315866929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:01:04.314179068 +0000 UTC m=+531.040137124" watchObservedRunningTime="2026-09-14 03:01:04.315866929 +0000 UTC m=+531.041824980"
Sep 14 03:01:04.642174 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:04.642079    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:04.646876 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:04.646853    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:05.298326 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.298294    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:05.299282 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.299261    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:01:05.767474 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.767444    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:01:05.769749 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.769728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz"
Sep 14 03:01:05.771849 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.771815    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 14 03:01:05.783574 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.783550    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:01:05.815241 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.815215    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:01:05.850602 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.850577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxn2q\" (UniqueName: \"kubernetes.io/projected/0f00d0aa-4331-4c73-aa5f-553943c9ada2-kube-api-access-pxn2q\") pod \"limitador-limitador-84994f7ffc-grwpz\" (UID: \"0f00d0aa-4331-4c73-aa5f-553943c9ada2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz"
Sep 14 03:01:05.850695 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.850615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0f00d0aa-4331-4c73-aa5f-553943c9ada2-config-file\") pod \"limitador-limitador-84994f7ffc-grwpz\" (UID: \"0f00d0aa-4331-4c73-aa5f-553943c9ada2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz"
Sep 14 03:01:05.951876 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.951853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxn2q\" (UniqueName: \"kubernetes.io/projected/0f00d0aa-4331-4c73-aa5f-553943c9ada2-kube-api-access-pxn2q\") pod \"limitador-limitador-84994f7ffc-grwpz\" (UID: \"0f00d0aa-4331-4c73-aa5f-553943c9ada2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz"
Sep 14 03:01:05.951979 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.951885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0f00d0aa-4331-4c73-aa5f-553943c9ada2-config-file\") pod \"limitador-limitador-84994f7ffc-grwpz\" (UID: \"0f00d0aa-4331-4c73-aa5f-553943c9ada2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz"
Sep 14 03:01:05.952455 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.952438    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0f00d0aa-4331-4c73-aa5f-553943c9ada2-config-file\") pod \"limitador-limitador-84994f7ffc-grwpz\" (UID: \"0f00d0aa-4331-4c73-aa5f-553943c9ada2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz"
Sep 14 03:01:05.960693 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:05.960670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxn2q\" (UniqueName: \"kubernetes.io/projected/0f00d0aa-4331-4c73-aa5f-553943c9ada2-kube-api-access-pxn2q\") pod \"limitador-limitador-84994f7ffc-grwpz\" (UID: \"0f00d0aa-4331-4c73-aa5f-553943c9ada2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz"
Sep 14 03:01:06.079730 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:06.079656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz"
Sep 14 03:01:06.352628 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:06.352601    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:01:06.353535 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:01:06.353511    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f00d0aa_4331_4c73_aa5f_553943c9ada2.slice/crio-39017b4374c08bb724d00eb21970cede3285ae3d222b094c09e1e5b3f7bb43c5 WatchSource:0}: Error finding container 39017b4374c08bb724d00eb21970cede3285ae3d222b094c09e1e5b3f7bb43c5: Status 404 returned error can't find the container with id 39017b4374c08bb724d00eb21970cede3285ae3d222b094c09e1e5b3f7bb43c5
Sep 14 03:01:07.310744 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:07.310687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz" event={"ID":"0f00d0aa-4331-4c73-aa5f-553943c9ada2","Type":"ContainerStarted","Data":"39017b4374c08bb724d00eb21970cede3285ae3d222b094c09e1e5b3f7bb43c5"}
Sep 14 03:01:08.316903 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:08.316869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz" event={"ID":"0f00d0aa-4331-4c73-aa5f-553943c9ada2","Type":"ContainerStarted","Data":"6160b6ef89e4ad28028bc5d6263960edfebcfeed21bf3c0968e2f30c495b9f03"}
Sep 14 03:01:08.317399 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:08.316997    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz"
Sep 14 03:01:08.335656 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:08.335609    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz" podStartSLOduration=2.203293074 podStartE2EDuration="3.335594043s" totalImagesPullingTime="1.132300969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:01:08.333323046 +0000 UTC m=+535.059281130" watchObservedRunningTime="2026-09-14 03:01:08.335594043 +0000 UTC m=+535.061552095"
Sep 14 03:01:19.323828 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:01:19.323794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-grwpz"
Sep 14 03:02:13.707423 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:13.707392    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:02:13.711186 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:13.711158    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:02:18.275847 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:18.275806    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:02:27.712002 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:27.711905    2580 generic.go:356] "Generic (PLEG): container finished" podID="65744834-d235-4b52-b655-6b0c87e69c27" containerID="4aa9d674ed9dc332888eac188230b38f0ee9893e94e861ffe4798aea64e9fa43" exitCode=0
Sep 14 03:02:27.712002 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:27.711980    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6" event={"ID":"65744834-d235-4b52-b655-6b0c87e69c27","Type":"ContainerDied","Data":"4aa9d674ed9dc332888eac188230b38f0ee9893e94e861ffe4798aea64e9fa43"}
Sep 14 03:02:27.712440 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:27.712331    2580 scope.go:117] "RemoveContainer" containerID="4aa9d674ed9dc332888eac188230b38f0ee9893e94e861ffe4798aea64e9fa43"
Sep 14 03:02:27.713267 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:27.713240    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:02:27.713336 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:27.713307    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:02:27.713381 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:27.713339    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:02:28.721902 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:28.721867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6" event={"ID":"65744834-d235-4b52-b655-6b0c87e69c27","Type":"ContainerStarted","Data":"ac12db49fa36eb8e61ea7bfda14e56b84a4e1194f6cbf30664e461b589d571aa"}
Sep 14 03:02:29.642903 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:29.642862    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:02:29.643091 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:29.643064    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-bh7f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 14 03:02:29.643142 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:29.643108    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6" podUID="65744834-d235-4b52-b655-6b0c87e69c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 14 03:02:30.643132 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:30.643094    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-bh7f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 14 03:02:30.643502 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:30.643164    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6" podUID="65744834-d235-4b52-b655-6b0c87e69c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 14 03:02:31.643041 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:31.643006    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-bh7f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 14 03:02:31.643221 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:31.643080    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6" podUID="65744834-d235-4b52-b655-6b0c87e69c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 14 03:02:32.642499 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:32.642461    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-bh7f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 14 03:02:32.642680 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:32.642519    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6" podUID="65744834-d235-4b52-b655-6b0c87e69c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 14 03:02:33.643212 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:33.643168    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-bh7f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 14 03:02:33.643593 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:33.643236    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6" podUID="65744834-d235-4b52-b655-6b0c87e69c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 14 03:02:34.596258 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:34.596220    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:02:34.646218 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:34.646189    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:02:34.646583 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:34.646534    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:02:34.647111 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:34.647097    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-bh7f6"
Sep 14 03:02:49.358738 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:49.358703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:02:58.561884 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:02:58.561846    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:03:04.562911 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:04.562872    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:03:20.162390 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:20.162359    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:03:31.858996 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.858965    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"]
Sep 14 03:03:31.861679 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.861659    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:31.864738 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.864716    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wdpm7\""
Sep 14 03:03:31.865004 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.864983    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 14 03:03:31.865105 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.865046    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 14 03:03:31.865105 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.865090    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 14 03:03:31.877243 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.877222    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"]
Sep 14 03:03:31.903569 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.903549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:31.903659 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.903579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405ebb2d-d206-43d7-a359-8130282f0a6c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:31.903659 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.903611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:31.903750 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.903732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8n8t\" (UniqueName: \"kubernetes.io/projected/405ebb2d-d206-43d7-a359-8130282f0a6c-kube-api-access-t8n8t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:31.903787 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.903768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:31.903823 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:31.903792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.004547 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.004512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8n8t\" (UniqueName: \"kubernetes.io/projected/405ebb2d-d206-43d7-a359-8130282f0a6c-kube-api-access-t8n8t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.004724 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.004556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.004724 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.004577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.004724 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.004606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.004724 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.004635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405ebb2d-d206-43d7-a359-8130282f0a6c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.004724 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.004674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.005088 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.004993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.005088 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.004999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.005170 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.005125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.007043 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.007003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405ebb2d-d206-43d7-a359-8130282f0a6c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.007206 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.007191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405ebb2d-d206-43d7-a359-8130282f0a6c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.015252 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.015222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8n8t\" (UniqueName: \"kubernetes.io/projected/405ebb2d-d206-43d7-a359-8130282f0a6c-kube-api-access-t8n8t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r9fpv\" (UID: \"405ebb2d-d206-43d7-a359-8130282f0a6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.172098 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.172005    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:32.445468 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.445444    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"]
Sep 14 03:03:32.447666 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:03:32.447639    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405ebb2d_d206_43d7_a359_8130282f0a6c.slice/crio-01744e49da7bc9300cb1fcd396610dacb921eeecb007cbaf2bb205d1a2545e4b WatchSource:0}: Error finding container 01744e49da7bc9300cb1fcd396610dacb921eeecb007cbaf2bb205d1a2545e4b: Status 404 returned error can't find the container with id 01744e49da7bc9300cb1fcd396610dacb921eeecb007cbaf2bb205d1a2545e4b
Sep 14 03:03:32.448985 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:32.448968    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 03:03:33.049232 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:33.049194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv" event={"ID":"405ebb2d-d206-43d7-a359-8130282f0a6c","Type":"ContainerStarted","Data":"01744e49da7bc9300cb1fcd396610dacb921eeecb007cbaf2bb205d1a2545e4b"}
Sep 14 03:03:33.766668 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:33.766641    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:03:34.055999 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:34.055916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv" event={"ID":"405ebb2d-d206-43d7-a359-8130282f0a6c","Type":"ContainerStarted","Data":"696be3879084cb0d73ed2e36b3f03af9336cd1c0f91a6722413b90a5969ba7d1"}
Sep 14 03:03:34.076808 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:34.076762    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv" podStartSLOduration=2.154803729 podStartE2EDuration="3.076747223s" totalImagesPullingTime="921.943494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:03:34.075256888 +0000 UTC m=+680.801214940" watchObservedRunningTime="2026-09-14 03:03:34.076747223 +0000 UTC m=+680.802705274"
Sep 14 03:03:42.172865 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.172827    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:42.173282 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.172886    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:42.185303 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.185282    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:42.863790 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.863761    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"]
Sep 14 03:03:42.866242 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.866227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:42.870255 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.870233    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 14 03:03:42.880620 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.880601    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"]
Sep 14 03:03:42.995062 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.995019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:42.995211 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.995065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:42.995211 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.995123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2th6h\" (UniqueName: \"kubernetes.io/projected/17dde9f1-556f-49f9-a355-dbe8826bbc19-kube-api-access-2th6h\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:42.995211 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.995176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:42.995211 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.995206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:42.995371 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:42.995257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dde9f1-556f-49f9-a355-dbe8826bbc19-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.096100 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.096071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dde9f1-556f-49f9-a355-dbe8826bbc19-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.096282 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.096119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.096282 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.096137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.096282 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.096167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2th6h\" (UniqueName: \"kubernetes.io/projected/17dde9f1-556f-49f9-a355-dbe8826bbc19-kube-api-access-2th6h\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.096282 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.096220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.096282 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.096247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.096590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.096558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.096650 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.096635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.096705 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.096685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.098631 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.098610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dde9f1-556f-49f9-a355-dbe8826bbc19-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.098742 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.098684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dde9f1-556f-49f9-a355-dbe8826bbc19-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.103878 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.103852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2th6h\" (UniqueName: \"kubernetes.io/projected/17dde9f1-556f-49f9-a355-dbe8826bbc19-kube-api-access-2th6h\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf\" (UID: \"17dde9f1-556f-49f9-a355-dbe8826bbc19\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.122025 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.121977    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r9fpv"
Sep 14 03:03:43.175833 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.175807    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:43.449158 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:43.449133    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"]
Sep 14 03:03:43.450310 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:03:43.450275    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17dde9f1_556f_49f9_a355_dbe8826bbc19.slice/crio-6ca91e81610f00218a30692d0a254ecc246cce1051d9800398a5eb42c185b882 WatchSource:0}: Error finding container 6ca91e81610f00218a30692d0a254ecc246cce1051d9800398a5eb42c185b882: Status 404 returned error can't find the container with id 6ca91e81610f00218a30692d0a254ecc246cce1051d9800398a5eb42c185b882
Sep 14 03:03:44.114170 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:44.114134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf" event={"ID":"17dde9f1-556f-49f9-a355-dbe8826bbc19","Type":"ContainerStarted","Data":"091457bfd7b7b28dcbee1076d222b869e107867b72d5b0b36d3c2b881363e363"}
Sep 14 03:03:44.114320 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:44.114180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf" event={"ID":"17dde9f1-556f-49f9-a355-dbe8826bbc19","Type":"ContainerStarted","Data":"6ca91e81610f00218a30692d0a254ecc246cce1051d9800398a5eb42c185b882"}
Sep 14 03:03:44.135726 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:44.135680    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf" podStartSLOduration=1.968884375 podStartE2EDuration="2.135664454s" totalImagesPullingTime="166.780079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:03:44.133239199 +0000 UTC m=+690.859197249" watchObservedRunningTime="2026-09-14 03:03:44.135664454 +0000 UTC m=+690.861622531"
Sep 14 03:03:45.354385 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:45.354352    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:03:53.176871 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:53.176838    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:53.177267 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:53.176892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:53.188848 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:53.188823    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:03:54.180123 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:03:54.180092    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf"
Sep 14 03:05:01.156372 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.156334    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"]
Sep 14 03:05:01.158911 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.158889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.165709 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.165686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9c9f1e-openshift-default-dockercfg-49bhd\""
Sep 14 03:05:01.206553 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.206527    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"]
Sep 14 03:05:01.218041 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.218005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-socket\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.218143 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.218050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-data\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.218143 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.218078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istiod-ca-cert\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.218143 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.218108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlgqp\" (UniqueName: \"kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-kube-api-access-xlgqp\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.218270 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.218153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-token\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.218270 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.218192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-envoy\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.218270 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.218219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-credential-socket\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.218270 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.218247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-certs\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.218270 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.218264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-podinfo\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.318841 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.318815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-certs\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.318959 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.318846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-podinfo\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.318959 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.318870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-socket\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.318959 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.318884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-data\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.318959 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.318901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istiod-ca-cert\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.318959 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.318927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlgqp\" (UniqueName: \"kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-kube-api-access-xlgqp\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.318959 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.318955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-token\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.319299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.318974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-envoy\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.319299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.318998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-credential-socket\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.319299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.319227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-data\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.319418 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.319292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-socket\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.319418 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.319313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-credential-socket\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.319543 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.319525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-certs\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.319803 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.319784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istiod-ca-cert\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.321333 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.321315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-envoy\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.321567 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.321550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-podinfo\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.352261 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.352231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlgqp\" (UniqueName: \"kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-kube-api-access-xlgqp\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.359130 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.359094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-token\") pod \"e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") " pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.468919 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.468830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:01.549865 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.549835    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"]
Sep 14 03:05:01.552545 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.552517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.556644 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.556620    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f002a1-openshift-default-dockercfg-8t5ps\""
Sep 14 03:05:01.575763 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.575742    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"]
Sep 14 03:05:01.622340 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.622308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-envoy\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.622519 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.622342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26b794a-4741-47f9-9773-68b2731d4bee-istio-podinfo\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.622519 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.622378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-credential-socket\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.622519 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.622446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-data\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.622519 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.622495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-socket\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.622685 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.622529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-istio-token\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.622685 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.622554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-certs\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.622685 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.622635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26b794a-4741-47f9-9773-68b2731d4bee-istiod-ca-cert\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.622782 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.622693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqmw2\" (UniqueName: \"kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-kube-api-access-xqmw2\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724219 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqmw2\" (UniqueName: \"kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-kube-api-access-xqmw2\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724219 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-envoy\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724219 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26b794a-4741-47f9-9773-68b2731d4bee-istio-podinfo\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724493 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-credential-socket\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724493 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-data\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724493 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-socket\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724493 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-istio-token\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724493 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-certs\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724493 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26b794a-4741-47f9-9773-68b2731d4bee-istiod-ca-cert\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724846 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-data\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724920 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-credential-socket\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.724979 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.724949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-socket\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.725182 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.725156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-certs\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.725302 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.725286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26b794a-4741-47f9-9773-68b2731d4bee-istiod-ca-cert\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.726573 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.726553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-envoy\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.726720 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.726703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26b794a-4741-47f9-9773-68b2731d4bee-istio-podinfo\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.734994 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.734972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-istio-token\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.735145 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.735129    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqmw2\" (UniqueName: \"kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-kube-api-access-xqmw2\") pod \"e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") " pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:01.754790 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:01.754768    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcd7270_0cdf_4370_b4e9_9b7a36354bc2.slice/crio-18d8ef696abf8b984aa24ed499e2542326bfa2cc8e2ce8e220e062acb27a3132 WatchSource:0}: Error finding container 18d8ef696abf8b984aa24ed499e2542326bfa2cc8e2ce8e220e062acb27a3132: Status 404 returned error can't find the container with id 18d8ef696abf8b984aa24ed499e2542326bfa2cc8e2ce8e220e062acb27a3132
Sep 14 03:05:01.757421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.757382    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:01.757498 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.757464    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:01.757537 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.757503    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:01.758404 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.758387    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"]
Sep 14 03:05:01.864117 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:01.864086    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:02.177641 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.177616    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"]
Sep 14 03:05:02.178504 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:02.178477    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26b794a_4741_47f9_9773_68b2731d4bee.slice/crio-9f8a4f80d59c69ba6861d4dc2c3c20d14ac2c464423edcc1983e64cfde13c9ad WatchSource:0}: Error finding container 9f8a4f80d59c69ba6861d4dc2c3c20d14ac2c464423edcc1983e64cfde13c9ad: Status 404 returned error can't find the container with id 9f8a4f80d59c69ba6861d4dc2c3c20d14ac2c464423edcc1983e64cfde13c9ad
Sep 14 03:05:02.181345 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.181310    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:02.181425 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.181385    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:02.181425 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.181416    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:02.516476 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.516429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" event={"ID":"d26b794a-4741-47f9-9773-68b2731d4bee","Type":"ContainerStarted","Data":"182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a"}
Sep 14 03:05:02.516661 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.516502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" event={"ID":"d26b794a-4741-47f9-9773-68b2731d4bee","Type":"ContainerStarted","Data":"9f8a4f80d59c69ba6861d4dc2c3c20d14ac2c464423edcc1983e64cfde13c9ad"}
Sep 14 03:05:02.518094 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.518069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" event={"ID":"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2","Type":"ContainerStarted","Data":"e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a"}
Sep 14 03:05:02.518224 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.518106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" event={"ID":"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2","Type":"ContainerStarted","Data":"18d8ef696abf8b984aa24ed499e2542326bfa2cc8e2ce8e220e062acb27a3132"}
Sep 14 03:05:02.562602 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.562548    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" podStartSLOduration=1.5625320249999999 podStartE2EDuration="1.562532025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:02.561696157 +0000 UTC m=+769.287654209" watchObservedRunningTime="2026-09-14 03:05:02.562532025 +0000 UTC m=+769.288490070"
Sep 14 03:05:02.624243 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.624194    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" podStartSLOduration=1.624177397 podStartE2EDuration="1.624177397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:02.622982052 +0000 UTC m=+769.348940103" watchObservedRunningTime="2026-09-14 03:05:02.624177397 +0000 UTC m=+769.350135448"
Sep 14 03:05:02.804200 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.804115    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7683-125b12"]
Sep 14 03:05:02.806721 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.806705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-125b12"
Sep 14 03:05:02.815209 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.815190    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2qhmh\""
Sep 14 03:05:02.830215 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.830192    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7683-125b12"]
Sep 14 03:05:02.865150 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.865124    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:02.937751 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:02.937720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9pqc\" (UniqueName: \"kubernetes.io/projected/78e7f469-ade6-4f73-9ec1-9964e7936ec8-kube-api-access-t9pqc\") pod \"test-curl-7683-125b12\" (UID: \"78e7f469-ade6-4f73-9ec1-9964e7936ec8\") " pod="opendatahub/test-curl-7683-125b12"
Sep 14 03:05:03.038807 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:03.038772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9pqc\" (UniqueName: \"kubernetes.io/projected/78e7f469-ade6-4f73-9ec1-9964e7936ec8-kube-api-access-t9pqc\") pod \"test-curl-7683-125b12\" (UID: \"78e7f469-ade6-4f73-9ec1-9964e7936ec8\") " pod="opendatahub/test-curl-7683-125b12"
Sep 14 03:05:03.054520 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:03.054440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9pqc\" (UniqueName: \"kubernetes.io/projected/78e7f469-ade6-4f73-9ec1-9964e7936ec8-kube-api-access-t9pqc\") pod \"test-curl-7683-125b12\" (UID: \"78e7f469-ade6-4f73-9ec1-9964e7936ec8\") " pod="opendatahub/test-curl-7683-125b12"
Sep 14 03:05:03.116046 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:03.116001    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-125b12"
Sep 14 03:05:03.396837 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:03.396737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7683-125b12"]
Sep 14 03:05:03.408779 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:03.408748    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78e7f469_ade6_4f73_9ec1_9964e7936ec8.slice/crio-735522ca90e3a64ccbab654eeb63cbd2eef2519102c1d2726e9e0a5913c26ad0 WatchSource:0}: Error finding container 735522ca90e3a64ccbab654eeb63cbd2eef2519102c1d2726e9e0a5913c26ad0: Status 404 returned error can't find the container with id 735522ca90e3a64ccbab654eeb63cbd2eef2519102c1d2726e9e0a5913c26ad0
Sep 14 03:05:03.469248 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:03.469214    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:03.474115 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:03.474090    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:03.524452 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:03.524418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7683-125b12" event={"ID":"78e7f469-ade6-4f73-9ec1-9964e7936ec8","Type":"ContainerStarted","Data":"735522ca90e3a64ccbab654eeb63cbd2eef2519102c1d2726e9e0a5913c26ad0"}
Sep 14 03:05:03.524645 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:03.524623    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:03.525620 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:03.525600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:03.869072 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:03.869045    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:04.533713 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:04.533674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7683-125b12" event={"ID":"78e7f469-ade6-4f73-9ec1-9964e7936ec8","Type":"ContainerStarted","Data":"7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0"}
Sep 14 03:05:04.534271 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:04.534253    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:04.535331 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:04.535309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:04.553892 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:04.553842    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7683-125b12" podStartSLOduration=1.9057524460000002 podStartE2EDuration="2.55382725s" totalImagesPullingTime="648.074804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:04.551191322 +0000 UTC m=+771.277149374" watchObservedRunningTime="2026-09-14 03:05:04.55382725 +0000 UTC m=+771.279785346"
Sep 14 03:05:05.437887 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:05.437846    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:05:05.479511 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:05.479432    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7683-125b12"]
Sep 14 03:05:05.481605 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:05.481583    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7683-125b12"]
Sep 14 03:05:05.539553 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:05:05.539525    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7683-125b12" container="test-curl-7683-125b12"
Sep 14 03:05:06.552160 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:06.552072    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7683-125b12" podUID="78e7f469-ade6-4f73-9ec1-9964e7936ec8" containerName="test-curl-7683-125b12" containerID="cri-o://7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0" gracePeriod=2
Sep 14 03:05:08.726825 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:08.726801    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-125b12"
Sep 14 03:05:08.796258 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:08.796226    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9pqc\" (UniqueName: \"kubernetes.io/projected/78e7f469-ade6-4f73-9ec1-9964e7936ec8-kube-api-access-t9pqc\") pod \"78e7f469-ade6-4f73-9ec1-9964e7936ec8\" (UID: \"78e7f469-ade6-4f73-9ec1-9964e7936ec8\") "
Sep 14 03:05:08.798381 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:08.798353    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e7f469-ade6-4f73-9ec1-9964e7936ec8-kube-api-access-t9pqc" (OuterVolumeSpecName: "kube-api-access-t9pqc") pod "78e7f469-ade6-4f73-9ec1-9964e7936ec8" (UID: "78e7f469-ade6-4f73-9ec1-9964e7936ec8"). InnerVolumeSpecName "kube-api-access-t9pqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:05:08.897926 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:08.897841    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9pqc\" (UniqueName: \"kubernetes.io/projected/78e7f469-ade6-4f73-9ec1-9964e7936ec8-kube-api-access-t9pqc\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:05:09.050866 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.050834    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"]
Sep 14 03:05:09.051250 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.051234    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e7f469-ade6-4f73-9ec1-9964e7936ec8" containerName="test-curl-7683-125b12"
Sep 14 03:05:09.051250 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.051249    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e7f469-ade6-4f73-9ec1-9964e7936ec8" containerName="test-curl-7683-125b12"
Sep 14 03:05:09.051416 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.051318    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e7f469-ade6-4f73-9ec1-9964e7936ec8" containerName="test-curl-7683-125b12"
Sep 14 03:05:09.056049 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.056016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:05:09.058666 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.058643    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f002a1-dockercfg-w2bxv\""
Sep 14 03:05:09.058772 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.058675    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f002a1\""
Sep 14 03:05:09.075636 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.075614    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"]
Sep 14 03:05:09.201518 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.201444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xz4\" (UniqueName: \"kubernetes.io/projected/763ec406-383b-458d-af55-2b45a337efa5-kube-api-access-p4xz4\") pod \"payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx\" (UID: \"763ec406-383b-458d-af55-2b45a337efa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:05:09.201518 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.201503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ec406-383b-458d-af55-2b45a337efa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx\" (UID: \"763ec406-383b-458d-af55-2b45a337efa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:05:09.302807 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.302773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xz4\" (UniqueName: \"kubernetes.io/projected/763ec406-383b-458d-af55-2b45a337efa5-kube-api-access-p4xz4\") pod \"payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx\" (UID: \"763ec406-383b-458d-af55-2b45a337efa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:05:09.302968 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.302850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ec406-383b-458d-af55-2b45a337efa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx\" (UID: \"763ec406-383b-458d-af55-2b45a337efa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:05:09.303512 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.303494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ec406-383b-458d-af55-2b45a337efa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx\" (UID: \"763ec406-383b-458d-af55-2b45a337efa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:05:09.314004 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.313977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xz4\" (UniqueName: \"kubernetes.io/projected/763ec406-383b-458d-af55-2b45a337efa5-kube-api-access-p4xz4\") pod \"payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx\" (UID: \"763ec406-383b-458d-af55-2b45a337efa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:05:09.366458 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.366435    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:05:09.568439 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.568392    2580 generic.go:356] "Generic (PLEG): container finished" podID="78e7f469-ade6-4f73-9ec1-9964e7936ec8" containerID="7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0" exitCode=137
Sep 14 03:05:09.568690 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.568583    2580 scope.go:117] "RemoveContainer" containerID="7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0"
Sep 14 03:05:09.568690 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.568678    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-125b12"
Sep 14 03:05:09.581295 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.581271    2580 scope.go:117] "RemoveContainer" containerID="7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0"
Sep 14 03:05:09.581400 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:05:09.581354    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7683-125b12\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="78e7f469-ade6-4f73-9ec1-9964e7936ec8" pod="opendatahub/test-curl-7683-125b12"
Sep 14 03:05:09.581564 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:05:09.581548    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0\": container with ID starting with 7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0 not found: ID does not exist" containerID="7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0"
Sep 14 03:05:09.581607 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.581572    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0"} err="failed to get container status \"7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0\": rpc error: code = NotFound desc = could not find container \"7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0\": container with ID starting with 7f78ca3b394fd4c4ce1330fdf5599ec15f07fdeedfc48d21d66e78d7424c64d0 not found: ID does not exist"
Sep 14 03:05:09.637311 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.637286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"]
Sep 14 03:05:09.639446 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:09.639419    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763ec406_383b_458d_af55_2b45a337efa5.slice/crio-793aaa062828d06c5a00cf4e5263022f8c8ae722fa26367c27c3a059669a8609 WatchSource:0}: Error finding container 793aaa062828d06c5a00cf4e5263022f8c8ae722fa26367c27c3a059669a8609: Status 404 returned error can't find the container with id 793aaa062828d06c5a00cf4e5263022f8c8ae722fa26367c27c3a059669a8609
Sep 14 03:05:09.756418 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:09.756394    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78e7f469-ade6-4f73-9ec1-9964e7936ec8" path="/var/lib/kubelet/pods/78e7f469-ade6-4f73-9ec1-9964e7936ec8/volumes"
Sep 14 03:05:10.546594 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.546491    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"]
Sep 14 03:05:10.549590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.549566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.553524 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.553497    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-ab5750-openshift-default-dockercfg-qtsn4\""
Sep 14 03:05:10.578069 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.578011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx" event={"ID":"763ec406-383b-458d-af55-2b45a337efa5","Type":"ContainerStarted","Data":"793aaa062828d06c5a00cf4e5263022f8c8ae722fa26367c27c3a059669a8609"}
Sep 14 03:05:10.615371 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.615311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-token\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.615654 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.615622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-data\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.615779 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.615672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dqht\" (UniqueName: \"kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-kube-api-access-6dqht\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.615779 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.615716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-credential-socket\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.615779 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.615738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-envoy\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.615945 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.615800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-socket\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.615945 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.615875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-podinfo\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.615945 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.615907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istiod-ca-cert\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.616129 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.615958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-certs\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.624581 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.624515    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"]
Sep 14 03:05:10.717459 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.717428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istiod-ca-cert\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.717632 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.717474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-certs\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.717632 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.717597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-token\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.717767 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.717657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-data\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.717829 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.717808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-certs\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.717894 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.717827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dqht\" (UniqueName: \"kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-kube-api-access-6dqht\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.717894 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.717883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-credential-socket\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.718007 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.717903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-envoy\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.718007 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.717970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-socket\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.718144 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.717899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-data\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.718144 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.718048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-podinfo\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.718267 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.718152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-credential-socket\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.718325 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.718278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istiod-ca-cert\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.718361 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.718331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-socket\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.719966 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.719939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-envoy\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.720399 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.720382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-podinfo\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.725684 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.725658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-token\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.726076 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.726058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dqht\" (UniqueName: \"kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-kube-api-access-6dqht\") pod \"e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") " pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:10.861919 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:10.861840    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:11.148041 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:11.148005    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"]
Sep 14 03:05:11.149367 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:11.149336    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84f7dd1_fb46_4476_badb_19eb0a7997f5.slice/crio-de0a95da74c1950f6a1029af4a803645c09a9c9dbf031d0e52afe3fcefc325b8 WatchSource:0}: Error finding container de0a95da74c1950f6a1029af4a803645c09a9c9dbf031d0e52afe3fcefc325b8: Status 404 returned error can't find the container with id de0a95da74c1950f6a1029af4a803645c09a9c9dbf031d0e52afe3fcefc325b8
Sep 14 03:05:11.151574 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:11.151542    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:11.151655 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:11.151605    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:11.151691 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:11.151652    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:11.584607 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:11.584573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx" event={"ID":"763ec406-383b-458d-af55-2b45a337efa5","Type":"ContainerStarted","Data":"68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be"}
Sep 14 03:05:11.584807 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:11.584639    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:05:11.586194 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:11.586166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" event={"ID":"c84f7dd1-fb46-4476-badb-19eb0a7997f5","Type":"ContainerStarted","Data":"22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0"}
Sep 14 03:05:11.586311 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:11.586210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" event={"ID":"c84f7dd1-fb46-4476-badb-19eb0a7997f5","Type":"ContainerStarted","Data":"de0a95da74c1950f6a1029af4a803645c09a9c9dbf031d0e52afe3fcefc325b8"}
Sep 14 03:05:11.606261 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:11.606217    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx" podStartSLOduration=2.51308168 podStartE2EDuration="3.606203149s" totalImagesPullingTime="1.093121469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:11.604515109 +0000 UTC m=+778.330473164" watchObservedRunningTime="2026-09-14 03:05:11.606203149 +0000 UTC m=+778.332161218"
Sep 14 03:05:11.637439 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:11.637397    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" podStartSLOduration=1.637371844 podStartE2EDuration="1.637371844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:11.633507607 +0000 UTC m=+778.359465649" watchObservedRunningTime="2026-09-14 03:05:11.637371844 +0000 UTC m=+778.363329896"
Sep 14 03:05:11.862886 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:11.862814    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:12.867924 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:12.867889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:13.598490 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:13.598457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:13.599433 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:13.599411    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:19.990811 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:19.990779    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"]
Sep 14 03:05:20.022051 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.021920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.023013 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.022983    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"]
Sep 14 03:05:20.024847 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.024772    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1493ac87-serving-cert\""
Sep 14 03:05:20.025056 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.024984    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:05:20.026914 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.025590    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jf78h\""
Sep 14 03:05:20.026914 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.025620    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 03:05:20.026914 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.025965    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 03:05:20.120316 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.120215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-maas-api-tls\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.120485 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.120401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a6daed-9804-4119-8b2e-11cae23061a4-openshift-service-ca\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.120485 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.120452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-metrics-certs\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.120604 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.120512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgrcr\" (UniqueName: \"kubernetes.io/projected/18a6daed-9804-4119-8b2e-11cae23061a4-kube-api-access-wgrcr\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.221628 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.221595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a6daed-9804-4119-8b2e-11cae23061a4-openshift-service-ca\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.221801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.221653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-metrics-certs\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.221801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.221686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgrcr\" (UniqueName: \"kubernetes.io/projected/18a6daed-9804-4119-8b2e-11cae23061a4-kube-api-access-wgrcr\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.221926 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.221803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-maas-api-tls\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.222462 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.222435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a6daed-9804-4119-8b2e-11cae23061a4-openshift-service-ca\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.224990 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.224965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-metrics-certs\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.225104 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.225086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-maas-api-tls\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.233019 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.232992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgrcr\" (UniqueName: \"kubernetes.io/projected/18a6daed-9804-4119-8b2e-11cae23061a4-kube-api-access-wgrcr\") pod \"maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.335663 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.335619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:20.624669 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.624631    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"]
Sep 14 03:05:20.626502 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:20.626474    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a6daed_9804_4119_8b2e_11cae23061a4.slice/crio-ad782181142a9601d5b077fd0dd69bb5533368e640221192d6138af29dbe3527 WatchSource:0}: Error finding container ad782181142a9601d5b077fd0dd69bb5533368e640221192d6138af29dbe3527: Status 404 returned error can't find the container with id ad782181142a9601d5b077fd0dd69bb5533368e640221192d6138af29dbe3527
Sep 14 03:05:20.671045 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.670927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf" event={"ID":"18a6daed-9804-4119-8b2e-11cae23061a4","Type":"ContainerStarted","Data":"ad782181142a9601d5b077fd0dd69bb5533368e640221192d6138af29dbe3527"}
Sep 14 03:05:20.789305 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.789271    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"]
Sep 14 03:05:20.794054 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.794011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:20.796739 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.796712    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1493ac87\""
Sep 14 03:05:20.797010 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.796988    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1493ac87-dockercfg-ck2tk\""
Sep 14 03:05:20.837066 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.836917    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"]
Sep 14 03:05:20.930458 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.930250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlvkp\" (UniqueName: \"kubernetes.io/projected/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-kube-api-access-hlvkp\") pod \"payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z\" (UID: \"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1\") " pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:20.930458 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.930372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-plugins-config-volume\") pod \"payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z\" (UID: \"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1\") " pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:20.948095 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.948060    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"]
Sep 14 03:05:20.951818 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:20.951796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:21.031760 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.031726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlvkp\" (UniqueName: \"kubernetes.io/projected/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-kube-api-access-hlvkp\") pod \"payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z\" (UID: \"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1\") " pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:21.032240 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.031813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-plugins-config-volume\") pod \"payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z\" (UID: \"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1\") " pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:21.032552 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.032530    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-plugins-config-volume\") pod \"payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z\" (UID: \"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1\") " pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:21.045949 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.045924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlvkp\" (UniqueName: \"kubernetes.io/projected/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-kube-api-access-hlvkp\") pod \"payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z\" (UID: \"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1\") " pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:21.108144 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.108101    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:21.132555 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.132512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xbbz\" (UniqueName: \"kubernetes.io/projected/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-kube-api-access-4xbbz\") pod \"payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9\" (UID: \"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:21.134306 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.133095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9\" (UID: \"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:21.203116 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.203009    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"]
Sep 14 03:05:21.236977 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.236938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xbbz\" (UniqueName: \"kubernetes.io/projected/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-kube-api-access-4xbbz\") pod \"payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9\" (UID: \"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:21.237175 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.236990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9\" (UID: \"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:21.237971 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.237941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9\" (UID: \"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:21.247447 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.247418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xbbz\" (UniqueName: \"kubernetes.io/projected/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-kube-api-access-4xbbz\") pod \"payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9\" (UID: \"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:21.262513 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.262366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:21.407492 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.407452    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"]
Sep 14 03:05:21.414694 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:21.414513    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1f0caf_ce09_4946_89ee_1d8580ecd6b1.slice/crio-40d6689ed8074402a31df488e7923cf3be11442af7935699c79fc894249834da WatchSource:0}: Error finding container 40d6689ed8074402a31df488e7923cf3be11442af7935699c79fc894249834da: Status 404 returned error can't find the container with id 40d6689ed8074402a31df488e7923cf3be11442af7935699c79fc894249834da
Sep 14 03:05:21.582539 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:21.582498    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae66e6ea_b0bc_492e_8a2a_262c9713d4bb.slice/crio-242e5fe2d7d01bf312e8171bb76efa666435c68cce12091c5fea9dcd33e8eb82 WatchSource:0}: Error finding container 242e5fe2d7d01bf312e8171bb76efa666435c68cce12091c5fea9dcd33e8eb82: Status 404 returned error can't find the container with id 242e5fe2d7d01bf312e8171bb76efa666435c68cce12091c5fea9dcd33e8eb82
Sep 14 03:05:21.582646 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.582613    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"]
Sep 14 03:05:21.684585 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.684483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9" event={"ID":"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb","Type":"ContainerStarted","Data":"242e5fe2d7d01bf312e8171bb76efa666435c68cce12091c5fea9dcd33e8eb82"}
Sep 14 03:05:21.689675 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.689530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z" event={"ID":"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1","Type":"ContainerStarted","Data":"c8cd99c9a8342e95d80044ff761363e102b659d9c25acdbb8032ce97091a76d9"}
Sep 14 03:05:21.689675 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.689587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z" event={"ID":"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1","Type":"ContainerStarted","Data":"40d6689ed8074402a31df488e7923cf3be11442af7935699c79fc894249834da"}
Sep 14 03:05:21.689675 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.689641    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:21.724097 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:21.722215    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z" podStartSLOduration=1.722193294 podStartE2EDuration="1.722193294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:21.720336813 +0000 UTC m=+788.446294867" watchObservedRunningTime="2026-09-14 03:05:21.722193294 +0000 UTC m=+788.448151348"
Sep 14 03:05:22.370851 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:22.370808    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:05:22.595242 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:22.594421    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:05:22.701303 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:22.701200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9" event={"ID":"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb","Type":"ContainerStarted","Data":"757b6258d9d626b8a5d223615112f663eea6f617d67f48043ca8964c52c7e092"}
Sep 14 03:05:22.701303 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:22.701290    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:22.788283 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:22.788223    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9" podStartSLOduration=2.788201606 podStartE2EDuration="2.788201606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:22.786888129 +0000 UTC m=+789.512846182" watchObservedRunningTime="2026-09-14 03:05:22.788201606 +0000 UTC m=+789.514159662"
Sep 14 03:05:24.719912 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.719871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf" event={"ID":"18a6daed-9804-4119-8b2e-11cae23061a4","Type":"ContainerStarted","Data":"3b7ce6cd71fb60eb880b6036d666cf34ea8ddc68d89292dc5bd2b7425d4df8f9"}
Sep 14 03:05:24.720806 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.720781    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:24.722641 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.722612    2580 generic.go:356] "Generic (PLEG): container finished" podID="d26b794a-4741-47f9-9773-68b2731d4bee" containerID="182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a" exitCode=0
Sep 14 03:05:24.722808 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.722788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" event={"ID":"d26b794a-4741-47f9-9773-68b2731d4bee","Type":"ContainerDied","Data":"182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a"}
Sep 14 03:05:24.723271 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.723251    2580 scope.go:117] "RemoveContainer" containerID="182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a"
Sep 14 03:05:24.725099 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.725059    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:24.725198 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.725154    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:24.725256 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.725207    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:24.730245 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.730215    2580 generic.go:356] "Generic (PLEG): container finished" podID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerID="e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a" exitCode=0
Sep 14 03:05:24.730360 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.730259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" event={"ID":"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2","Type":"ContainerDied","Data":"e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a"}
Sep 14 03:05:24.730873 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.730854    2580 scope.go:117] "RemoveContainer" containerID="e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a"
Sep 14 03:05:24.735941 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.735722    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:24.735941 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.735780    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:24.735941 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.735810    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:24.743377 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:24.743327    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf" podStartSLOduration=2.3470838450000002 podStartE2EDuration="5.743308995s" totalImagesPullingTime="3.39622515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:24.739431691 +0000 UTC m=+791.465389743" watchObservedRunningTime="2026-09-14 03:05:24.743308995 +0000 UTC m=+791.469267049"
Sep 14 03:05:25.740360 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:25.740322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" event={"ID":"d26b794a-4741-47f9-9773-68b2731d4bee","Type":"ContainerStarted","Data":"5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a"}
Sep 14 03:05:25.742536 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:25.742499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" event={"ID":"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2","Type":"ContainerStarted","Data":"c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b"}
Sep 14 03:05:25.864580 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:25.864495    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:25.865357 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:25.865098    2580 patch_prober.go:28] interesting pod/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 14 03:05:25.865357 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:25.865155    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 14 03:05:26.470095 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:26.470056    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:26.470439 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:26.470413    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 14 03:05:26.470491 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:26.470465    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 14 03:05:26.864765 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:26.864728    2580 patch_prober.go:28] interesting pod/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 14 03:05:26.865404 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:26.865372    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 14 03:05:27.470021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:27.469929    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 14 03:05:27.470021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:27.470094    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 14 03:05:27.865369 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:27.865329    2580 patch_prober.go:28] interesting pod/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 14 03:05:27.865948 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:27.865911    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 14 03:05:28.469749 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:28.469709    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 14 03:05:28.469957 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:28.469792    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 14 03:05:28.868151 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:28.868116    2580 patch_prober.go:28] interesting pod/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 14 03:05:28.868966 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:28.868936    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 14 03:05:29.469639 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:29.469600    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 14 03:05:29.469925 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:29.469896    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 14 03:05:29.864792 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:29.864758    2580 patch_prober.go:28] interesting pod/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 14 03:05:29.864973 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:29.864811    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 14 03:05:30.469967 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:30.469923    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 14 03:05:30.470460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:30.469984    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 14 03:05:30.864655 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:30.864618    2580 patch_prober.go:28] interesting pod/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 14 03:05:30.864960 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:30.864925    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 14 03:05:31.470091 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:31.470050    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 14 03:05:31.470564 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:31.470105    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 14 03:05:31.760900 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:31.760870    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:31.865209 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:31.865158    2580 patch_prober.go:28] interesting pod/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 14 03:05:31.865350 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:31.865239    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 14 03:05:32.474456 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:32.474427    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:32.474862 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:32.474842    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:32.475558 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:32.475537    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:05:32.703721 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:32.703696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:32.869782 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:32.869754    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:32.870053 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:32.870022    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:32.870720 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:32.870700    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:05:34.732707 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:34.732680    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:34.826718 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:34.826683    2580 generic.go:356] "Generic (PLEG): container finished" podID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerID="22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0" exitCode=0
Sep 14 03:05:34.826886 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:34.826747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" event={"ID":"c84f7dd1-fb46-4476-badb-19eb0a7997f5","Type":"ContainerDied","Data":"22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0"}
Sep 14 03:05:34.827364 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:34.827348    2580 scope.go:117] "RemoveContainer" containerID="22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0"
Sep 14 03:05:34.828582 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:34.828550    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:34.828682 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:34.828620    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:34.828682 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:34.828652    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:35.238317 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.238284    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"]
Sep 14 03:05:35.241097 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.241080    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"
Sep 14 03:05:35.243952 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.243933    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sv5sd\""
Sep 14 03:05:35.269846 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.269822    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"]
Sep 14 03:05:35.314292 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.314261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb7a500-face-4d8a-901f-8c90c12e8a56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5\" (UID: \"2bb7a500-face-4d8a-901f-8c90c12e8a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"
Sep 14 03:05:35.415635 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.415588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb7a500-face-4d8a-901f-8c90c12e8a56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5\" (UID: \"2bb7a500-face-4d8a-901f-8c90c12e8a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"
Sep 14 03:05:35.416499 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.416466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb7a500-face-4d8a-901f-8c90c12e8a56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5\" (UID: \"2bb7a500-face-4d8a-901f-8c90c12e8a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"
Sep 14 03:05:35.550798 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.550762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"
Sep 14 03:05:35.835711 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.835631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" event={"ID":"c84f7dd1-fb46-4476-badb-19eb0a7997f5","Type":"ContainerStarted","Data":"3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae"}
Sep 14 03:05:35.853174 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.853149    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"]
Sep 14 03:05:35.854630 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:35.854605    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb7a500_face_4d8a_901f_8c90c12e8a56.slice/crio-fe7a7259df6dd69ad2100c1c73f8c0e821adb6cbd6f7918604dd246c7a03d332 WatchSource:0}: Error finding container fe7a7259df6dd69ad2100c1c73f8c0e821adb6cbd6f7918604dd246c7a03d332: Status 404 returned error can't find the container with id fe7a7259df6dd69ad2100c1c73f8c0e821adb6cbd6f7918604dd246c7a03d332
Sep 14 03:05:35.862021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.862000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:35.862224 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.862195    2580 patch_prober.go:28] interesting pod/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 14 03:05:35.862280 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:35.862239    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 14 03:05:36.849052 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:36.848993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5" event={"ID":"2bb7a500-face-4d8a-901f-8c90c12e8a56","Type":"ContainerStarted","Data":"fe7a7259df6dd69ad2100c1c73f8c0e821adb6cbd6f7918604dd246c7a03d332"}
Sep 14 03:05:36.862207 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:36.862182    2580 patch_prober.go:28] interesting pod/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 14 03:05:36.862272 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:36.862229    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 14 03:05:37.862518 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:37.862488    2580 patch_prober.go:28] interesting pod/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 14 03:05:37.862872 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:37.862549    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 14 03:05:38.862800 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:38.862768    2580 patch_prober.go:28] interesting pod/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 14 03:05:38.863137 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:38.862817    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 14 03:05:39.862205 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:39.862171    2580 patch_prober.go:28] interesting pod/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 14 03:05:39.862398 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:39.862225    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 14 03:05:39.868818 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:39.868794    2580 generic.go:356] "Generic (PLEG): container finished" podID="2bb7a500-face-4d8a-901f-8c90c12e8a56" containerID="8835d1f319a21d6b3275d860bc33a9ada0c50d33088eaf105486d7ad40905371" exitCode=0
Sep 14 03:05:39.869179 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:39.868870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5" event={"ID":"2bb7a500-face-4d8a-901f-8c90c12e8a56","Type":"ContainerDied","Data":"8835d1f319a21d6b3275d860bc33a9ada0c50d33088eaf105486d7ad40905371"}
Sep 14 03:05:40.863118 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:40.863082    2580 patch_prober.go:28] interesting pod/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 14 03:05:40.863321 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:40.863156    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 14 03:05:41.016230 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:41.016207    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"
Sep 14 03:05:41.074155 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:41.074127    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb7a500-face-4d8a-901f-8c90c12e8a56-maas-api-service-ca\") pod \"2bb7a500-face-4d8a-901f-8c90c12e8a56\" (UID: \"2bb7a500-face-4d8a-901f-8c90c12e8a56\") "
Sep 14 03:05:41.074502 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:41.074481    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb7a500-face-4d8a-901f-8c90c12e8a56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bb7a500-face-4d8a-901f-8c90c12e8a56" (UID: "2bb7a500-face-4d8a-901f-8c90c12e8a56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:05:41.174881 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:41.174821    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb7a500-face-4d8a-901f-8c90c12e8a56-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:05:41.866924 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:41.866893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:41.867199 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:41.867181    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:41.867945 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:41.867929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:05:41.880513 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:41.880486    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5" event={"ID":"2bb7a500-face-4d8a-901f-8c90c12e8a56","Type":"ContainerDied","Data":"fe7a7259df6dd69ad2100c1c73f8c0e821adb6cbd6f7918604dd246c7a03d332"}
Sep 14 03:05:41.880513 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:41.880504    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"
Sep 14 03:05:41.880758 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:41.880527    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe7a7259df6dd69ad2100c1c73f8c0e821adb6cbd6f7918604dd246c7a03d332"
Sep 14 03:05:42.130564 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:42.130468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:05:48.660775 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.660736    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"]
Sep 14 03:05:48.661229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.660986    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf" podUID="18a6daed-9804-4119-8b2e-11cae23061a4" containerName="maas-api" containerID="cri-o://3b7ce6cd71fb60eb880b6036d666cf34ea8ddc68d89292dc5bd2b7425d4df8f9" gracePeriod=30
Sep 14 03:05:48.720751 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.720716    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"]
Sep 14 03:05:48.721016 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.720988    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z" podUID="9f1f0caf-ce09-4946-89ee-1d8580ecd6b1" containerName="payload-processing" containerID="cri-o://c8cd99c9a8342e95d80044ff761363e102b659d9c25acdbb8032ce97091a76d9" gracePeriod=30
Sep 14 03:05:48.730434 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.730408    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"]
Sep 14 03:05:48.730720 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.730670    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9" podUID="ae66e6ea-b0bc-492e-8a2a-262c9713d4bb" containerName="payload-pre-processing" containerID="cri-o://757b6258d9d626b8a5d223615112f663eea6f617d67f48043ca8964c52c7e092" gracePeriod=30
Sep 14 03:05:48.902048 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.900080    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"]
Sep 14 03:05:48.902048 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.901059    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1493ac87-rb4h5"]
Sep 14 03:05:48.923362 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.923290    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f1f0caf-ce09-4946-89ee-1d8580ecd6b1" containerID="c8cd99c9a8342e95d80044ff761363e102b659d9c25acdbb8032ce97091a76d9" exitCode=0
Sep 14 03:05:48.923551 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.923366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z" event={"ID":"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1","Type":"ContainerDied","Data":"c8cd99c9a8342e95d80044ff761363e102b659d9c25acdbb8032ce97091a76d9"}
Sep 14 03:05:48.924927 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.924904    2580 generic.go:356] "Generic (PLEG): container finished" podID="ae66e6ea-b0bc-492e-8a2a-262c9713d4bb" containerID="757b6258d9d626b8a5d223615112f663eea6f617d67f48043ca8964c52c7e092" exitCode=0
Sep 14 03:05:48.925041 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.924980    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9" event={"ID":"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb","Type":"ContainerDied","Data":"757b6258d9d626b8a5d223615112f663eea6f617d67f48043ca8964c52c7e092"}
Sep 14 03:05:48.926471 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.926449    2580 generic.go:356] "Generic (PLEG): container finished" podID="18a6daed-9804-4119-8b2e-11cae23061a4" containerID="3b7ce6cd71fb60eb880b6036d666cf34ea8ddc68d89292dc5bd2b7425d4df8f9" exitCode=0
Sep 14 03:05:48.926554 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:48.926487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf" event={"ID":"18a6daed-9804-4119-8b2e-11cae23061a4","Type":"ContainerDied","Data":"3b7ce6cd71fb60eb880b6036d666cf34ea8ddc68d89292dc5bd2b7425d4df8f9"}
Sep 14 03:05:49.038758 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.038725    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:49.048214 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.048185    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:49.064435 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.064411    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:49.150087 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-maas-api-tls\") pod \"18a6daed-9804-4119-8b2e-11cae23061a4\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") "
Sep 14 03:05:49.150245 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150116    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a6daed-9804-4119-8b2e-11cae23061a4-openshift-service-ca\") pod \"18a6daed-9804-4119-8b2e-11cae23061a4\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") "
Sep 14 03:05:49.150245 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150156    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlvkp\" (UniqueName: \"kubernetes.io/projected/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-kube-api-access-hlvkp\") pod \"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1\" (UID: \"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1\") "
Sep 14 03:05:49.150245 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150202    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-plugins-config-volume\") pod \"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb\" (UID: \"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb\") "
Sep 14 03:05:49.150376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150258    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgrcr\" (UniqueName: \"kubernetes.io/projected/18a6daed-9804-4119-8b2e-11cae23061a4-kube-api-access-wgrcr\") pod \"18a6daed-9804-4119-8b2e-11cae23061a4\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") "
Sep 14 03:05:49.150376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150311    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xbbz\" (UniqueName: \"kubernetes.io/projected/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-kube-api-access-4xbbz\") pod \"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb\" (UID: \"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb\") "
Sep 14 03:05:49.150376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150336    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-metrics-certs\") pod \"18a6daed-9804-4119-8b2e-11cae23061a4\" (UID: \"18a6daed-9804-4119-8b2e-11cae23061a4\") "
Sep 14 03:05:49.150376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150369    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-plugins-config-volume\") pod \"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1\" (UID: \"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1\") "
Sep 14 03:05:49.150589 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150454    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a6daed-9804-4119-8b2e-11cae23061a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18a6daed-9804-4119-8b2e-11cae23061a4" (UID: "18a6daed-9804-4119-8b2e-11cae23061a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:05:49.150589 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150575    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae66e6ea-b0bc-492e-8a2a-262c9713d4bb" (UID: "ae66e6ea-b0bc-492e-8a2a-262c9713d4bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:05:49.150885 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150859    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a6daed-9804-4119-8b2e-11cae23061a4-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:05:49.151001 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150889    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:05:49.151001 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.150891    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f1f0caf-ce09-4946-89ee-1d8580ecd6b1" (UID: "9f1f0caf-ce09-4946-89ee-1d8580ecd6b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:05:49.152586 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.152554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-kube-api-access-4xbbz" (OuterVolumeSpecName: "kube-api-access-4xbbz") pod "ae66e6ea-b0bc-492e-8a2a-262c9713d4bb" (UID: "ae66e6ea-b0bc-492e-8a2a-262c9713d4bb"). InnerVolumeSpecName "kube-api-access-4xbbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:05:49.152699 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.152586    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a6daed-9804-4119-8b2e-11cae23061a4-kube-api-access-wgrcr" (OuterVolumeSpecName: "kube-api-access-wgrcr") pod "18a6daed-9804-4119-8b2e-11cae23061a4" (UID: "18a6daed-9804-4119-8b2e-11cae23061a4"). InnerVolumeSpecName "kube-api-access-wgrcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:05:49.152699 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.152595    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-kube-api-access-hlvkp" (OuterVolumeSpecName: "kube-api-access-hlvkp") pod "9f1f0caf-ce09-4946-89ee-1d8580ecd6b1" (UID: "9f1f0caf-ce09-4946-89ee-1d8580ecd6b1"). InnerVolumeSpecName "kube-api-access-hlvkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:05:49.152803 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.152701    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18a6daed-9804-4119-8b2e-11cae23061a4" (UID: "18a6daed-9804-4119-8b2e-11cae23061a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:05:49.152803 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.152731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "18a6daed-9804-4119-8b2e-11cae23061a4" (UID: "18a6daed-9804-4119-8b2e-11cae23061a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:05:49.252453 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.252371    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgrcr\" (UniqueName: \"kubernetes.io/projected/18a6daed-9804-4119-8b2e-11cae23061a4-kube-api-access-wgrcr\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:05:49.252453 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.252415    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xbbz\" (UniqueName: \"kubernetes.io/projected/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb-kube-api-access-4xbbz\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:05:49.252453 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.252426    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:05:49.252453 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.252435    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:05:49.252453 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.252442    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a6daed-9804-4119-8b2e-11cae23061a4-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:05:49.252453 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.252451    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlvkp\" (UniqueName: \"kubernetes.io/projected/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1-kube-api-access-hlvkp\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:05:49.756999 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.756967    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb7a500-face-4d8a-901f-8c90c12e8a56" path="/var/lib/kubelet/pods/2bb7a500-face-4d8a-901f-8c90c12e8a56/volumes"
Sep 14 03:05:49.933291 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.933223    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"
Sep 14 03:05:49.933291 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.933261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf" event={"ID":"18a6daed-9804-4119-8b2e-11cae23061a4","Type":"ContainerDied","Data":"ad782181142a9601d5b077fd0dd69bb5533368e640221192d6138af29dbe3527"}
Sep 14 03:05:49.933519 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.933318    2580 scope.go:117] "RemoveContainer" containerID="3b7ce6cd71fb60eb880b6036d666cf34ea8ddc68d89292dc5bd2b7425d4df8f9"
Sep 14 03:05:49.935240 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.935220    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"
Sep 14 03:05:49.935240 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.935232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z" event={"ID":"9f1f0caf-ce09-4946-89ee-1d8580ecd6b1","Type":"ContainerDied","Data":"40d6689ed8074402a31df488e7923cf3be11442af7935699c79fc894249834da"}
Sep 14 03:05:49.936826 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.936805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9" event={"ID":"ae66e6ea-b0bc-492e-8a2a-262c9713d4bb","Type":"ContainerDied","Data":"242e5fe2d7d01bf312e8171bb76efa666435c68cce12091c5fea9dcd33e8eb82"}
Sep 14 03:05:49.936826 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.936824    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"
Sep 14 03:05:49.944364 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.944347    2580 scope.go:117] "RemoveContainer" containerID="c8cd99c9a8342e95d80044ff761363e102b659d9c25acdbb8032ce97091a76d9"
Sep 14 03:05:49.952485 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.952467    2580 scope.go:117] "RemoveContainer" containerID="757b6258d9d626b8a5d223615112f663eea6f617d67f48043ca8964c52c7e092"
Sep 14 03:05:49.956191 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.956171    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"]
Sep 14 03:05:49.968640 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:49.968621    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1493ac87-76f8b7766c-jsj6z"]
Sep 14 03:05:50.002234 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.002213    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"]
Sep 14 03:05:50.021223 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.021202    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1493ac87-7d885b7ff6-j6vbf"]
Sep 14 03:05:50.040875 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.040855    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"]
Sep 14 03:05:50.055622 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.055601    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1493ac87-75c7f9845d-ltwk9"]
Sep 14 03:05:50.108701 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.108678    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"]
Sep 14 03:05:50.109051 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109018    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a6daed-9804-4119-8b2e-11cae23061a4" containerName="maas-api"
Sep 14 03:05:50.109051 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109045    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a6daed-9804-4119-8b2e-11cae23061a4" containerName="maas-api"
Sep 14 03:05:50.109133 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109060    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae66e6ea-b0bc-492e-8a2a-262c9713d4bb" containerName="payload-pre-processing"
Sep 14 03:05:50.109133 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109069    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae66e6ea-b0bc-492e-8a2a-262c9713d4bb" containerName="payload-pre-processing"
Sep 14 03:05:50.109133 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109087    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f1f0caf-ce09-4946-89ee-1d8580ecd6b1" containerName="payload-processing"
Sep 14 03:05:50.109133 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109095    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f1f0caf-ce09-4946-89ee-1d8580ecd6b1" containerName="payload-processing"
Sep 14 03:05:50.109133 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109108    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb7a500-face-4d8a-901f-8c90c12e8a56" containerName="revoke-keys"
Sep 14 03:05:50.109133 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109113    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb7a500-face-4d8a-901f-8c90c12e8a56" containerName="revoke-keys"
Sep 14 03:05:50.109312 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109170    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb7a500-face-4d8a-901f-8c90c12e8a56" containerName="revoke-keys"
Sep 14 03:05:50.109312 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109178    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a6daed-9804-4119-8b2e-11cae23061a4" containerName="maas-api"
Sep 14 03:05:50.109312 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109185    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f1f0caf-ce09-4946-89ee-1d8580ecd6b1" containerName="payload-processing"
Sep 14 03:05:50.109312 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.109193    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae66e6ea-b0bc-492e-8a2a-262c9713d4bb" containerName="payload-pre-processing"
Sep 14 03:05:50.113719 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.113703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.116462 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.116442    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d8a641aa-gw-openshift-default-dockercfg-srxkl\""
Sep 14 03:05:50.181411 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.181380    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"]
Sep 14 03:05:50.259336 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.259309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-certs\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.259503 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.259350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-credential-socket\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.259503 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.259428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e309690-02b8-4d64-8aa9-1002580cf5b6-istiod-ca-cert\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.259600 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.259518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-data\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.259600 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.259543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-podinfo\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.259600 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.259572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-envoy\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.259714 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.259589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-token\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.259714 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.259658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zgv\" (UniqueName: \"kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-kube-api-access-l4zgv\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.259714 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.259696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-socket\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.360505 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.360480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-data\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.360505 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.360508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-podinfo\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.360702 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.360527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-envoy\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.360702 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.360544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-token\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.360702 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.360593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zgv\" (UniqueName: \"kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-kube-api-access-l4zgv\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.360702 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.360637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-socket\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.360702 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.360662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-certs\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.360702 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.360684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-credential-socket\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.361009 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.360769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e309690-02b8-4d64-8aa9-1002580cf5b6-istiod-ca-cert\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.361009 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.360984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-data\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.361143 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.361063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-socket\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.361143 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.361100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-certs\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.361249 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.361215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-credential-socket\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.361418 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.361399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e309690-02b8-4d64-8aa9-1002580cf5b6-istiod-ca-cert\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.363012 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.362996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-envoy\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.363245 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.363229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-podinfo\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.369616 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.369594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zgv\" (UniqueName: \"kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-kube-api-access-l4zgv\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.370339 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.370315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-token\") pod \"e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") " pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.423811 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.423789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:50.703767 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.703716    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"]
Sep 14 03:05:50.705739 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:50.705692    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e309690_02b8_4d64_8aa9_1002580cf5b6.slice/crio-671b449486c513bd2551f562389be7e7a9ccffce5799210917f3a41b7eed185c WatchSource:0}: Error finding container 671b449486c513bd2551f562389be7e7a9ccffce5799210917f3a41b7eed185c: Status 404 returned error can't find the container with id 671b449486c513bd2551f562389be7e7a9ccffce5799210917f3a41b7eed185c
Sep 14 03:05:50.708607 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.708574    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:50.708706 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.708638    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:50.708706 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.708664    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:05:50.948807 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.948766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" event={"ID":"9e309690-02b8-4d64-8aa9-1002580cf5b6","Type":"ContainerStarted","Data":"ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3"}
Sep 14 03:05:50.949284 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.948826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" event={"ID":"9e309690-02b8-4d64-8aa9-1002580cf5b6","Type":"ContainerStarted","Data":"671b449486c513bd2551f562389be7e7a9ccffce5799210917f3a41b7eed185c"}
Sep 14 03:05:50.998587 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:50.998488    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" podStartSLOduration=0.998469197 podStartE2EDuration="998.469197ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:50.995454436 +0000 UTC m=+817.721412488" watchObservedRunningTime="2026-09-14 03:05:50.998469197 +0000 UTC m=+817.724427250"
Sep 14 03:05:51.424421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:51.424387    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:51.430227 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:51.430190    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:51.757742 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:51.757710    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a6daed-9804-4119-8b2e-11cae23061a4" path="/var/lib/kubelet/pods/18a6daed-9804-4119-8b2e-11cae23061a4/volumes"
Sep 14 03:05:51.758129 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:51.758115    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f1f0caf-ce09-4946-89ee-1d8580ecd6b1" path="/var/lib/kubelet/pods/9f1f0caf-ce09-4946-89ee-1d8580ecd6b1/volumes"
Sep 14 03:05:51.758441 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:51.758429    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae66e6ea-b0bc-492e-8a2a-262c9713d4bb" path="/var/lib/kubelet/pods/ae66e6ea-b0bc-492e-8a2a-262c9713d4bb/volumes"
Sep 14 03:05:51.954665 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:51.954633    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:51.955738 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:51.955714    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:05:54.645591 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.645561    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"]
Sep 14 03:05:54.650286 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.650265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.652627 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.652607    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d8a641aa-serving-cert\""
Sep 14 03:05:54.652882 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.652862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jf78h\""
Sep 14 03:05:54.652991 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.652908    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 03:05:54.652991 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.652920    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 03:05:54.652991 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.652927    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:05:54.661223 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.661205    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"]
Sep 14 03:05:54.804557 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.804527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmz2\" (UniqueName: \"kubernetes.io/projected/af6315f0-f348-4bdf-9dab-7eedcc6198f4-kube-api-access-skmz2\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.804737 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.804583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-maas-api-tls\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.804737 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.804639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-metrics-certs\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.804866 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.804793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af6315f0-f348-4bdf-9dab-7eedcc6198f4-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.905890 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.905802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af6315f0-f348-4bdf-9dab-7eedcc6198f4-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.906084 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.906025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmz2\" (UniqueName: \"kubernetes.io/projected/af6315f0-f348-4bdf-9dab-7eedcc6198f4-kube-api-access-skmz2\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.906170 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.906119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-maas-api-tls\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.906170 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.906144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-metrics-certs\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.906555 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.906532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af6315f0-f348-4bdf-9dab-7eedcc6198f4-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.908489 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.908467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-metrics-certs\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.908568 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.908520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-maas-api-tls\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.918907 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.918880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmz2\" (UniqueName: \"kubernetes.io/projected/af6315f0-f348-4bdf-9dab-7eedcc6198f4-kube-api-access-skmz2\") pod \"maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:54.961699 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:54.961675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:55.247441 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.247417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"]
Sep 14 03:05:55.248107 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:55.248078    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6315f0_f348_4bdf_9dab_7eedcc6198f4.slice/crio-ea6e728931a6d7d010961eedb22d8c9da5e8a80454395aec5858fd42e0fbc06b WatchSource:0}: Error finding container ea6e728931a6d7d010961eedb22d8c9da5e8a80454395aec5858fd42e0fbc06b: Status 404 returned error can't find the container with id ea6e728931a6d7d010961eedb22d8c9da5e8a80454395aec5858fd42e0fbc06b
Sep 14 03:05:55.416611 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.416575    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"]
Sep 14 03:05:55.421368 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.421346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:05:55.423880 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.423862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d8a641aa-dockercfg-w82zw\""
Sep 14 03:05:55.423999 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.423899    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d8a641aa\""
Sep 14 03:05:55.429856 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.429692    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"]
Sep 14 03:05:55.511432 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.511395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79knn\" (UniqueName: \"kubernetes.io/projected/dff5fed0-0d64-4956-8b14-b762943c474f-kube-api-access-79knn\") pod \"payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7\" (UID: \"dff5fed0-0d64-4956-8b14-b762943c474f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:05:55.511599 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.511450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff5fed0-0d64-4956-8b14-b762943c474f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7\" (UID: \"dff5fed0-0d64-4956-8b14-b762943c474f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:05:55.612475 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.612443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79knn\" (UniqueName: \"kubernetes.io/projected/dff5fed0-0d64-4956-8b14-b762943c474f-kube-api-access-79knn\") pod \"payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7\" (UID: \"dff5fed0-0d64-4956-8b14-b762943c474f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:05:55.612642 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.612509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff5fed0-0d64-4956-8b14-b762943c474f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7\" (UID: \"dff5fed0-0d64-4956-8b14-b762943c474f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:05:55.613258 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.613233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff5fed0-0d64-4956-8b14-b762943c474f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7\" (UID: \"dff5fed0-0d64-4956-8b14-b762943c474f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:05:55.621756 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.621738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79knn\" (UniqueName: \"kubernetes.io/projected/dff5fed0-0d64-4956-8b14-b762943c474f-kube-api-access-79knn\") pod \"payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7\" (UID: \"dff5fed0-0d64-4956-8b14-b762943c474f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:05:55.734209 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.734183    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:05:55.982157 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.982126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9" event={"ID":"af6315f0-f348-4bdf-9dab-7eedcc6198f4","Type":"ContainerStarted","Data":"32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9"}
Sep 14 03:05:55.982314 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.982176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9" event={"ID":"af6315f0-f348-4bdf-9dab-7eedcc6198f4","Type":"ContainerStarted","Data":"ea6e728931a6d7d010961eedb22d8c9da5e8a80454395aec5858fd42e0fbc06b"}
Sep 14 03:05:55.982314 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:55.982251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:05:56.007695 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.007647    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9" podStartSLOduration=1.6188639729999998 podStartE2EDuration="2.007630968s" totalImagesPullingTime="388.766995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:56.006540248 +0000 UTC m=+822.732498316" watchObservedRunningTime="2026-09-14 03:05:56.007630968 +0000 UTC m=+822.733589019"
Sep 14 03:05:56.014439 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.014417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"]
Sep 14 03:05:56.016616 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:56.016591    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff5fed0_0d64_4956_8b14_b762943c474f.slice/crio-913c9d5bcfed8225bef6cddd830afd644c98104bb52e2edb125e53b054120360 WatchSource:0}: Error finding container 913c9d5bcfed8225bef6cddd830afd644c98104bb52e2edb125e53b054120360: Status 404 returned error can't find the container with id 913c9d5bcfed8225bef6cddd830afd644c98104bb52e2edb125e53b054120360
Sep 14 03:05:56.746350 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.746316    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"]
Sep 14 03:05:56.749903 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.749886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.752729 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.752708    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"default-dockercfg-5qjfv\""
Sep 14 03:05:56.752854 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.752827    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"kube-root-ca.crt\""
Sep 14 03:05:56.752924 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.752827    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"simulator-w4-9c9f1e-kserve-self-signed-certs\""
Sep 14 03:05:56.752981 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.752921    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"openshift-service-ca.crt\""
Sep 14 03:05:56.759666 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.759645    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"]
Sep 14 03:05:56.824800 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.824771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c500aa2-1218-430f-9e91-0325ec0a08de-tls-certs\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.824917 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.824816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4v6q\" (UniqueName: \"kubernetes.io/projected/0c500aa2-1218-430f-9e91-0325ec0a08de-kube-api-access-k4v6q\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.824917 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.824876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-dshm\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.825007 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.824922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-model-cache\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.825007 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.824957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-home\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.825115 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.825012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-tmp-dir\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.926146 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.926114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-dshm\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.926266 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.926157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-model-cache\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.926266 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.926191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-home\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.926266 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.926245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-tmp-dir\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.926381 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.926314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c500aa2-1218-430f-9e91-0325ec0a08de-tls-certs\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.926381 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.926351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4v6q\" (UniqueName: \"kubernetes.io/projected/0c500aa2-1218-430f-9e91-0325ec0a08de-kube-api-access-k4v6q\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.926629 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.926607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-model-cache\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.926711 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.926633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-home\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.926711 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.926674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-tmp-dir\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.928427 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.928410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-dshm\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.928860 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.928841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c500aa2-1218-430f-9e91-0325ec0a08de-tls-certs\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.934600 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.934577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4v6q\" (UniqueName: \"kubernetes.io/projected/0c500aa2-1218-430f-9e91-0325ec0a08de-kube-api-access-k4v6q\") pod \"simulator-w4-9c9f1e-kserve-86685774cd-clx2m\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") " pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:56.989630 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.989606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7" event={"ID":"dff5fed0-0d64-4956-8b14-b762943c474f","Type":"ContainerStarted","Data":"e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607"}
Sep 14 03:05:56.989745 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.989650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7" event={"ID":"dff5fed0-0d64-4956-8b14-b762943c474f","Type":"ContainerStarted","Data":"913c9d5bcfed8225bef6cddd830afd644c98104bb52e2edb125e53b054120360"}
Sep 14 03:05:56.989875 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:56.989858    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:05:57.013131 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:57.013087    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7" podStartSLOduration=2.013072316 podStartE2EDuration="2.013072316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:57.01083659 +0000 UTC m=+823.736794644" watchObservedRunningTime="2026-09-14 03:05:57.013072316 +0000 UTC m=+823.739030368"
Sep 14 03:05:57.059896 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:57.059874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:05:57.334780 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:57.334752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"]
Sep 14 03:05:57.335596 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:05:57.335575    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c500aa2_1218_430f_9e91_0325ec0a08de.slice/crio-03f6d07b920b76a01b3a03f565271e4a0988fada9f2fe90dfe39768ec6bc68e8 WatchSource:0}: Error finding container 03f6d07b920b76a01b3a03f565271e4a0988fada9f2fe90dfe39768ec6bc68e8: Status 404 returned error can't find the container with id 03f6d07b920b76a01b3a03f565271e4a0988fada9f2fe90dfe39768ec6bc68e8
Sep 14 03:05:57.998056 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:57.997988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m" event={"ID":"0c500aa2-1218-430f-9e91-0325ec0a08de","Type":"ContainerStarted","Data":"d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9"}
Sep 14 03:05:57.998056 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:57.998058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m" event={"ID":"0c500aa2-1218-430f-9e91-0325ec0a08de","Type":"ContainerStarted","Data":"03f6d07b920b76a01b3a03f565271e4a0988fada9f2fe90dfe39768ec6bc68e8"}
Sep 14 03:05:58.022060 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:58.022002    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m" podStartSLOduration=2.021988006 podStartE2EDuration="2.021988006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:58.019626085 +0000 UTC m=+824.745584138" watchObservedRunningTime="2026-09-14 03:05:58.021988006 +0000 UTC m=+824.747946059"
Sep 14 03:05:59.667711 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:05:59.667679    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:06:01.996759 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:01.996728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:06:07.060677 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:07.060647    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:06:07.061112 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:07.060723    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:06:07.074571 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:07.074540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:06:07.997222 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:07.997172    2580 patch_prober.go:28] interesting pod/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 03:06:07.997392 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:07.997287    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 03:06:08.000847 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:08.000827    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:06:08.072730 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:08.072702    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:06:10.075661 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:10.075627    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerID="ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3" exitCode=0
Sep 14 03:06:10.076088 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:10.075702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" event={"ID":"9e309690-02b8-4d64-8aa9-1002580cf5b6","Type":"ContainerDied","Data":"ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3"}
Sep 14 03:06:10.076228 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:10.076213    2580 scope.go:117] "RemoveContainer" containerID="ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3"
Sep 14 03:06:10.077277 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:10.077251    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:06:10.077330 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:10.077316    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:06:10.077369 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:10.077345    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:06:11.084123 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:11.084084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" event={"ID":"9e309690-02b8-4d64-8aa9-1002580cf5b6","Type":"ContainerStarted","Data":"dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41"}
Sep 14 03:06:11.425011 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:11.424920    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:06:11.425214 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:11.425175    2580 patch_prober.go:28] interesting pod/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 14 03:06:11.425293 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:11.425233    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 14 03:06:12.424276 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:12.424238    2580 patch_prober.go:28] interesting pod/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 14 03:06:12.424641 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:12.424299    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 14 03:06:13.425202 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:13.425167    2580 patch_prober.go:28] interesting pod/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 14 03:06:13.425555 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:13.425224    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 14 03:06:14.424161 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:14.424123    2580 patch_prober.go:28] interesting pod/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 14 03:06:14.424332 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:14.424181    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 14 03:06:14.660042 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:14.659996    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:06:15.424974 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:15.424940    2580 patch_prober.go:28] interesting pod/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 14 03:06:15.425164 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:15.424996    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 14 03:06:16.428746 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:16.428706    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:06:16.429143 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:16.429062    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:06:16.429958 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:16.429938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:06:17.559335 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.559299    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"]
Sep 14 03:06:17.566200 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.566177    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.569431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.569409    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a752f\"/\"simulator-w0-3a752f-kserve-self-signed-certs\""
Sep 14 03:06:17.569587 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.569409    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3a752f\"/\"kube-root-ca.crt\""
Sep 14 03:06:17.569668 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.569451    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3a752f\"/\"openshift-service-ca.crt\""
Sep 14 03:06:17.569668 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.569526    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a752f\"/\"default-dockercfg-w6bjn\""
Sep 14 03:06:17.572340 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.572318    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"]
Sep 14 03:06:17.653145 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.653116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-home\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.653287 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.653169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-tmp-dir\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.653287 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.653195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-model-cache\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.653287 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.653281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32150566-1bac-4eca-b69f-1b997aeefbbe-tls-certs\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.653412 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.653296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgtf5\" (UniqueName: \"kubernetes.io/projected/32150566-1bac-4eca-b69f-1b997aeefbbe-kube-api-access-bgtf5\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.653412 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.653326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-dshm\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.753965 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.753913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32150566-1bac-4eca-b69f-1b997aeefbbe-tls-certs\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.754188 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.753976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgtf5\" (UniqueName: \"kubernetes.io/projected/32150566-1bac-4eca-b69f-1b997aeefbbe-kube-api-access-bgtf5\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.754188 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.754073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-dshm\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.754188 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.754151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-home\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.754373 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.754205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-tmp-dir\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.754373 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.754231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-model-cache\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.754595 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.754570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-tmp-dir\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.754704 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.754598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-home\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.754704 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.754665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-model-cache\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.756456 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.756438    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-dshm\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.756698 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.756678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32150566-1bac-4eca-b69f-1b997aeefbbe-tls-certs\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.762182 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.762162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgtf5\" (UniqueName: \"kubernetes.io/projected/32150566-1bac-4eca-b69f-1b997aeefbbe-kube-api-access-bgtf5\") pod \"simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") " pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:17.876554 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:17.876477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:18.158020 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:18.157995    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"]
Sep 14 03:06:18.159004 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:06:18.158974    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32150566_1bac_4eca_b69f_1b997aeefbbe.slice/crio-fb2a47a91c754d1bb61503c995e94bda5ecbae12c658f983d3de449c9522abd4 WatchSource:0}: Error finding container fb2a47a91c754d1bb61503c995e94bda5ecbae12c658f983d3de449c9522abd4: Status 404 returned error can't find the container with id fb2a47a91c754d1bb61503c995e94bda5ecbae12c658f983d3de449c9522abd4
Sep 14 03:06:19.143130 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:19.143091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn" event={"ID":"32150566-1bac-4eca-b69f-1b997aeefbbe","Type":"ContainerStarted","Data":"c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a"}
Sep 14 03:06:19.143476 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:19.143145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn" event={"ID":"32150566-1bac-4eca-b69f-1b997aeefbbe","Type":"ContainerStarted","Data":"fb2a47a91c754d1bb61503c995e94bda5ecbae12c658f983d3de449c9522abd4"}
Sep 14 03:06:19.175472 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:19.175426    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn" podStartSLOduration=2.175411164 podStartE2EDuration="2.175411164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:06:19.174388603 +0000 UTC m=+845.900346657" watchObservedRunningTime="2026-09-14 03:06:19.175411164 +0000 UTC m=+845.901369215"
Sep 14 03:06:27.877163 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:27.877129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:27.877563 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:27.877176    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:27.889564 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:27.889539    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:28.205754 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:28.205677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:06:32.864313 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:32.864269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:06:34.441293 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.441260    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"]
Sep 14 03:06:34.441774 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.441442    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9" podUID="af6315f0-f348-4bdf-9dab-7eedcc6198f4" containerName="maas-api" containerID="cri-o://32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9" gracePeriod=30
Sep 14 03:06:34.507179 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.507146    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"]
Sep 14 03:06:34.507402 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.507376    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7" podUID="dff5fed0-0d64-4956-8b14-b762943c474f" containerName="payload-pre-processing" containerID="cri-o://e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607" gracePeriod=30
Sep 14 03:06:34.757866 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.757841    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:06:34.820689 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.820644    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:06:34.847742 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.847575    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"]
Sep 14 03:06:34.847926 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.847803    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy" containerID="cri-o://dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41" gracePeriod=30
Sep 14 03:06:34.931856 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.931824    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-metrics-certs\") pod \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") "
Sep 14 03:06:34.932009 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.931880    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff5fed0-0d64-4956-8b14-b762943c474f-plugins-config-volume\") pod \"dff5fed0-0d64-4956-8b14-b762943c474f\" (UID: \"dff5fed0-0d64-4956-8b14-b762943c474f\") "
Sep 14 03:06:34.932009 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.931906    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skmz2\" (UniqueName: \"kubernetes.io/projected/af6315f0-f348-4bdf-9dab-7eedcc6198f4-kube-api-access-skmz2\") pod \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") "
Sep 14 03:06:34.932139 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.932003    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-maas-api-tls\") pod \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") "
Sep 14 03:06:34.932194 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.932180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79knn\" (UniqueName: \"kubernetes.io/projected/dff5fed0-0d64-4956-8b14-b762943c474f-kube-api-access-79knn\") pod \"dff5fed0-0d64-4956-8b14-b762943c474f\" (UID: \"dff5fed0-0d64-4956-8b14-b762943c474f\") "
Sep 14 03:06:34.932248 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.932214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af6315f0-f348-4bdf-9dab-7eedcc6198f4-openshift-service-ca\") pod \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\" (UID: \"af6315f0-f348-4bdf-9dab-7eedcc6198f4\") "
Sep 14 03:06:34.932322 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.932289    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff5fed0-0d64-4956-8b14-b762943c474f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dff5fed0-0d64-4956-8b14-b762943c474f" (UID: "dff5fed0-0d64-4956-8b14-b762943c474f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:06:34.932645 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.932623    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff5fed0-0d64-4956-8b14-b762943c474f-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:34.932765 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.932641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6315f0-f348-4bdf-9dab-7eedcc6198f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af6315f0-f348-4bdf-9dab-7eedcc6198f4" (UID: "af6315f0-f348-4bdf-9dab-7eedcc6198f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:06:34.934002 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.933978    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "af6315f0-f348-4bdf-9dab-7eedcc6198f4" (UID: "af6315f0-f348-4bdf-9dab-7eedcc6198f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:06:34.934123 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.934092    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af6315f0-f348-4bdf-9dab-7eedcc6198f4" (UID: "af6315f0-f348-4bdf-9dab-7eedcc6198f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:06:34.934384 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.934368    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6315f0-f348-4bdf-9dab-7eedcc6198f4-kube-api-access-skmz2" (OuterVolumeSpecName: "kube-api-access-skmz2") pod "af6315f0-f348-4bdf-9dab-7eedcc6198f4" (UID: "af6315f0-f348-4bdf-9dab-7eedcc6198f4"). InnerVolumeSpecName "kube-api-access-skmz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:06:34.934445 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:34.934428    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff5fed0-0d64-4956-8b14-b762943c474f-kube-api-access-79knn" (OuterVolumeSpecName: "kube-api-access-79knn") pod "dff5fed0-0d64-4956-8b14-b762943c474f" (UID: "dff5fed0-0d64-4956-8b14-b762943c474f"). InnerVolumeSpecName "kube-api-access-79knn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:06:35.033926 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.033898    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79knn\" (UniqueName: \"kubernetes.io/projected/dff5fed0-0d64-4956-8b14-b762943c474f-kube-api-access-79knn\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:35.033926 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.033924    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af6315f0-f348-4bdf-9dab-7eedcc6198f4-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:35.034142 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.033934    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:35.034142 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.033943    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skmz2\" (UniqueName: \"kubernetes.io/projected/af6315f0-f348-4bdf-9dab-7eedcc6198f4-kube-api-access-skmz2\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:35.034142 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.033951    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af6315f0-f348-4bdf-9dab-7eedcc6198f4-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:35.233578 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.233541    2580 generic.go:356] "Generic (PLEG): container finished" podID="dff5fed0-0d64-4956-8b14-b762943c474f" containerID="e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607" exitCode=0
Sep 14 03:06:35.233768 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.233582    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"
Sep 14 03:06:35.233768 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.233620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7" event={"ID":"dff5fed0-0d64-4956-8b14-b762943c474f","Type":"ContainerDied","Data":"e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607"}
Sep 14 03:06:35.233768 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.233677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7" event={"ID":"dff5fed0-0d64-4956-8b14-b762943c474f","Type":"ContainerDied","Data":"913c9d5bcfed8225bef6cddd830afd644c98104bb52e2edb125e53b054120360"}
Sep 14 03:06:35.233768 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.233707    2580 scope.go:117] "RemoveContainer" containerID="e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607"
Sep 14 03:06:35.235219 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.235147    2580 generic.go:356] "Generic (PLEG): container finished" podID="af6315f0-f348-4bdf-9dab-7eedcc6198f4" containerID="32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9" exitCode=0
Sep 14 03:06:35.235317 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.235218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9" event={"ID":"af6315f0-f348-4bdf-9dab-7eedcc6198f4","Type":"ContainerDied","Data":"32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9"}
Sep 14 03:06:35.235317 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.235263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9" event={"ID":"af6315f0-f348-4bdf-9dab-7eedcc6198f4","Type":"ContainerDied","Data":"ea6e728931a6d7d010961eedb22d8c9da5e8a80454395aec5858fd42e0fbc06b"}
Sep 14 03:06:35.235317 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.235200    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"
Sep 14 03:06:35.243159 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.243139    2580 scope.go:117] "RemoveContainer" containerID="e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607"
Sep 14 03:06:35.243466 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:06:35.243446    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607\": container with ID starting with e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607 not found: ID does not exist" containerID="e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607"
Sep 14 03:06:35.243532 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.243473    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607"} err="failed to get container status \"e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607\": rpc error: code = NotFound desc = could not find container \"e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607\": container with ID starting with e44ada694f3775426055c31981d8886be029484c7324df6c579283e7f194c607 not found: ID does not exist"
Sep 14 03:06:35.243532 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.243490    2580 scope.go:117] "RemoveContainer" containerID="32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9"
Sep 14 03:06:35.251867 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.251842    2580 scope.go:117] "RemoveContainer" containerID="32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9"
Sep 14 03:06:35.252150 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:06:35.252131    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9\": container with ID starting with 32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9 not found: ID does not exist" containerID="32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9"
Sep 14 03:06:35.252216 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.252156    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9"} err="failed to get container status \"32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9\": rpc error: code = NotFound desc = could not find container \"32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9\": container with ID starting with 32fdc2810a0189bd01257e71d8dd574e1e9d45c433a5be3e17152ee32a3b11e9 not found: ID does not exist"
Sep 14 03:06:35.258276 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.258256    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"]
Sep 14 03:06:35.262935 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.262914    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8a641aa-57976ccd94-xw7m7"]
Sep 14 03:06:35.272989 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.272966    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"]
Sep 14 03:06:35.276855 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.276831    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8a641aa-6d98fd9547-592n9"]
Sep 14 03:06:35.616783 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.616754    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"]
Sep 14 03:06:35.617163 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.617149    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af6315f0-f348-4bdf-9dab-7eedcc6198f4" containerName="maas-api"
Sep 14 03:06:35.617163 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.617164    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af6315f0-f348-4bdf-9dab-7eedcc6198f4" containerName="maas-api"
Sep 14 03:06:35.617242 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.617179    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff5fed0-0d64-4956-8b14-b762943c474f" containerName="payload-pre-processing"
Sep 14 03:06:35.617242 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.617184    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff5fed0-0d64-4956-8b14-b762943c474f" containerName="payload-pre-processing"
Sep 14 03:06:35.617306 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.617248    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af6315f0-f348-4bdf-9dab-7eedcc6198f4" containerName="maas-api"
Sep 14 03:06:35.617306 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.617257    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff5fed0-0d64-4956-8b14-b762943c474f" containerName="payload-pre-processing"
Sep 14 03:06:35.621654 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.621636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.624403 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.624383    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-65e77699-openshift-default-dockercfg-pntps\""
Sep 14 03:06:35.634331 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.634311    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"]
Sep 14 03:06:35.741744 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.741635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddn9p\" (UniqueName: \"kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-kube-api-access-ddn9p\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.741744 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.741714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dd5875-18de-4353-9917-44887c30e52e-istio-podinfo\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.741993 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.741802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dd5875-18de-4353-9917-44887c30e52e-istiod-ca-cert\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.741993 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.741838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-certs\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.741993 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.741870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-socket\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.741993 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.741949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-envoy\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.741993 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.741982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-credential-socket\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.742304 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.742018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-istio-token\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.742304 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.742139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-data\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.757888 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.757862    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af6315f0-f348-4bdf-9dab-7eedcc6198f4" path="/var/lib/kubelet/pods/af6315f0-f348-4bdf-9dab-7eedcc6198f4/volumes"
Sep 14 03:06:35.758256 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.758241    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff5fed0-0d64-4956-8b14-b762943c474f" path="/var/lib/kubelet/pods/dff5fed0-0d64-4956-8b14-b762943c474f/volumes"
Sep 14 03:06:35.843022 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.842996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddn9p\" (UniqueName: \"kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-kube-api-access-ddn9p\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.843201 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.843092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dd5875-18de-4353-9917-44887c30e52e-istio-podinfo\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.843201 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.843169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dd5875-18de-4353-9917-44887c30e52e-istiod-ca-cert\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.843328 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.843201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-certs\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.843328 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.843226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-socket\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.843328 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.843298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-envoy\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.843495 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.843326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-credential-socket\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.843495 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.843349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-istio-token\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.843495 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.843378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-data\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.843794 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.843771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-data\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.845563 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.845513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dd5875-18de-4353-9917-44887c30e52e-istiod-ca-cert\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.845919 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.845899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-credential-socket\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.846181 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.846161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-socket\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.846412 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.846363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-certs\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.848855 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.848812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-envoy\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.849078 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.849058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dd5875-18de-4353-9917-44887c30e52e-istio-podinfo\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.851750 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.851725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-istio-token\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.852774 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.852756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddn9p\" (UniqueName: \"kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-kube-api-access-ddn9p\") pod \"e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") " pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:35.931835 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:35.931812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:36.207166 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:36.207140    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"]
Sep 14 03:06:36.208337 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:06:36.208311    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71dd5875_18de_4353_9917_44887c30e52e.slice/crio-693d8c3cfee81c4478c4a56d0165516f025ee85f477bf15cb90e6a6fa170870b WatchSource:0}: Error finding container 693d8c3cfee81c4478c4a56d0165516f025ee85f477bf15cb90e6a6fa170870b: Status 404 returned error can't find the container with id 693d8c3cfee81c4478c4a56d0165516f025ee85f477bf15cb90e6a6fa170870b
Sep 14 03:06:36.210644 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:36.210608    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:06:36.210748 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:36.210673    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:06:36.210748 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:36.210702    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:06:36.244083 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:36.244061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" event={"ID":"71dd5875-18de-4353-9917-44887c30e52e","Type":"ContainerStarted","Data":"693d8c3cfee81c4478c4a56d0165516f025ee85f477bf15cb90e6a6fa170870b"}
Sep 14 03:06:37.252634 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:37.252600    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" event={"ID":"71dd5875-18de-4353-9917-44887c30e52e","Type":"ContainerStarted","Data":"d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39"}
Sep 14 03:06:37.290699 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:37.290649    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" podStartSLOduration=2.290635402 podStartE2EDuration="2.290635402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:06:37.28859207 +0000 UTC m=+864.014550121" watchObservedRunningTime="2026-09-14 03:06:37.290635402 +0000 UTC m=+864.016593454"
Sep 14 03:06:37.932736 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:37.932704    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:37.937648 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:37.937627    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:38.258867 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:38.258838    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:38.259776 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:38.259759    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:06:40.102715 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.102692    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:06:40.185361 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185296    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-certs\") pod \"9e309690-02b8-4d64-8aa9-1002580cf5b6\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") "
Sep 14 03:06:40.185361 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-token\") pod \"9e309690-02b8-4d64-8aa9-1002580cf5b6\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") "
Sep 14 03:06:40.185515 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185389    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-socket\") pod \"9e309690-02b8-4d64-8aa9-1002580cf5b6\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") "
Sep 14 03:06:40.185515 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e309690-02b8-4d64-8aa9-1002580cf5b6-istiod-ca-cert\") pod \"9e309690-02b8-4d64-8aa9-1002580cf5b6\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") "
Sep 14 03:06:40.185515 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-envoy\") pod \"9e309690-02b8-4d64-8aa9-1002580cf5b6\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") "
Sep 14 03:06:40.185663 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185544    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e309690-02b8-4d64-8aa9-1002580cf5b6" (UID: "9e309690-02b8-4d64-8aa9-1002580cf5b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:06:40.185663 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185606    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-data\") pod \"9e309690-02b8-4d64-8aa9-1002580cf5b6\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") "
Sep 14 03:06:40.185663 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185658    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-credential-socket\") pod \"9e309690-02b8-4d64-8aa9-1002580cf5b6\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") "
Sep 14 03:06:40.185815 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e309690-02b8-4d64-8aa9-1002580cf5b6" (UID: "9e309690-02b8-4d64-8aa9-1002580cf5b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:06:40.185815 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185701    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-podinfo\") pod \"9e309690-02b8-4d64-8aa9-1002580cf5b6\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") "
Sep 14 03:06:40.185815 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4zgv\" (UniqueName: \"kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-kube-api-access-l4zgv\") pod \"9e309690-02b8-4d64-8aa9-1002580cf5b6\" (UID: \"9e309690-02b8-4d64-8aa9-1002580cf5b6\") "
Sep 14 03:06:40.185815 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185795    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e309690-02b8-4d64-8aa9-1002580cf5b6" (UID: "9e309690-02b8-4d64-8aa9-1002580cf5b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:06:40.186099 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e309690-02b8-4d64-8aa9-1002580cf5b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e309690-02b8-4d64-8aa9-1002580cf5b6" (UID: "9e309690-02b8-4d64-8aa9-1002580cf5b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:06:40.186099 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.185923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e309690-02b8-4d64-8aa9-1002580cf5b6" (UID: "9e309690-02b8-4d64-8aa9-1002580cf5b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:06:40.186214 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.186167    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:40.186214 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.186187    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:40.186214 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.186201    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:40.186214 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.186214    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:40.186417 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.186226    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e309690-02b8-4d64-8aa9-1002580cf5b6-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:40.187647 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.187626    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e309690-02b8-4d64-8aa9-1002580cf5b6" (UID: "9e309690-02b8-4d64-8aa9-1002580cf5b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:06:40.187756 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.187670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e309690-02b8-4d64-8aa9-1002580cf5b6" (UID: "9e309690-02b8-4d64-8aa9-1002580cf5b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:06:40.187807 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.187753    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e309690-02b8-4d64-8aa9-1002580cf5b6" (UID: "9e309690-02b8-4d64-8aa9-1002580cf5b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 03:06:40.187807 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.187799    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-kube-api-access-l4zgv" (OuterVolumeSpecName: "kube-api-access-l4zgv") pod "9e309690-02b8-4d64-8aa9-1002580cf5b6" (UID: "9e309690-02b8-4d64-8aa9-1002580cf5b6"). InnerVolumeSpecName "kube-api-access-l4zgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:06:40.273466 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.273441    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerID="dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41" exitCode=0
Sep 14 03:06:40.273587 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.273521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" event={"ID":"9e309690-02b8-4d64-8aa9-1002580cf5b6","Type":"ContainerDied","Data":"dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41"}
Sep 14 03:06:40.273587 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.273533    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"
Sep 14 03:06:40.273587 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.273578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5" event={"ID":"9e309690-02b8-4d64-8aa9-1002580cf5b6","Type":"ContainerDied","Data":"671b449486c513bd2551f562389be7e7a9ccffce5799210917f3a41b7eed185c"}
Sep 14 03:06:40.273698 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.273611    2580 scope.go:117] "RemoveContainer" containerID="dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41"
Sep 14 03:06:40.283155 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.283134    2580 scope.go:117] "RemoveContainer" containerID="ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3"
Sep 14 03:06:40.287621 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.287599    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:40.287751 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.287625    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:40.287751 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.287638    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4zgv\" (UniqueName: \"kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-kube-api-access-l4zgv\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:40.287751 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.287650    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e309690-02b8-4d64-8aa9-1002580cf5b6-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:06:40.291652 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.291636    2580 scope.go:117] "RemoveContainer" containerID="dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41"
Sep 14 03:06:40.291920 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:06:40.291901    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41\": container with ID starting with dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41 not found: ID does not exist" containerID="dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41"
Sep 14 03:06:40.291961 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.291929    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41"} err="failed to get container status \"dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41\": rpc error: code = NotFound desc = could not find container \"dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41\": container with ID starting with dfb5fd4b7f06721bc5939c874ed0fd2c42683c2f2d33175a624025496aa73a41 not found: ID does not exist"
Sep 14 03:06:40.291961 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.291944    2580 scope.go:117] "RemoveContainer" containerID="ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3"
Sep 14 03:06:40.292223 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:06:40.292204    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3\": container with ID starting with ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3 not found: ID does not exist" containerID="ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3"
Sep 14 03:06:40.292304 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.292229    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3"} err="failed to get container status \"ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3\": rpc error: code = NotFound desc = could not find container \"ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3\": container with ID starting with ed45244758b21685a6f64aefcffdd7e7849d3a1437cd0fd13a9715ed23a55ab3 not found: ID does not exist"
Sep 14 03:06:40.308046 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.308006    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"]
Sep 14 03:06:40.314966 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.314946    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d8a641aa-gw-openshift-default-7fc8c44d7b-tqdd5"]
Sep 14 03:06:40.317409 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.317384    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"]
Sep 14 03:06:40.317773 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.317759    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy"
Sep 14 03:06:40.317850 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.317774    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy"
Sep 14 03:06:40.317850 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.317784    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy"
Sep 14 03:06:40.317850 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.317791    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy"
Sep 14 03:06:40.318011 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.317869    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy"
Sep 14 03:06:40.318011 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.317880    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" containerName="istio-proxy"
Sep 14 03:06:40.322313 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.322295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.324938 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.324728    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 03:06:40.324938 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.324753    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jf78h\""
Sep 14 03:06:40.324938 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.324768    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 03:06:40.324938 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.324937    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-65e77699-serving-cert\""
Sep 14 03:06:40.325226 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.325022    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:06:40.327602 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.327581    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"]
Sep 14 03:06:40.388587 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.388558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfwsp\" (UniqueName: \"kubernetes.io/projected/4669ed30-1ae0-487e-ac93-9f17e6cccf64-kube-api-access-xfwsp\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.388688 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.388598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-metrics-certs\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.388688 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.388667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-maas-api-tls\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.388789 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.388725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4669ed30-1ae0-487e-ac93-9f17e6cccf64-openshift-service-ca\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.447160 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.447082    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"]
Sep 14 03:06:40.450502 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.450487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.452640 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.452617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ab5750\"/\"default-dockercfg-w6hqj\""
Sep 14 03:06:40.452774 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.452638    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ab5750\"/\"openshift-service-ca.crt\""
Sep 14 03:06:40.452774 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.452686    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ab5750\"/\"kube-root-ca.crt\""
Sep 14 03:06:40.452997 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.452983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ab5750\"/\"autores-model-ab5750-kserve-self-signed-certs\""
Sep 14 03:06:40.463636 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.463612    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"]
Sep 14 03:06:40.489450 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.489430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-home\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.489548 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.489472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfwsp\" (UniqueName: \"kubernetes.io/projected/4669ed30-1ae0-487e-ac93-9f17e6cccf64-kube-api-access-xfwsp\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.489548 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.489495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-metrics-certs\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.489655 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.489558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-model-cache\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.489655 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.489604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-maas-api-tls\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.489655 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.489626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86nbw\" (UniqueName: \"kubernetes.io/projected/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-kube-api-access-86nbw\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.489655 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.489644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tmp-dir\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.489825 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.489673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-dshm\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.489825 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:06:40.489720    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-serving-cert: secret "maas-api-e2e-ait-65e77699-serving-cert" not found
Sep 14 03:06:40.489825 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:06:40.489783    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-maas-api-tls podName:4669ed30-1ae0-487e-ac93-9f17e6cccf64 nodeName:}" failed. No retries permitted until 2026-09-14 03:06:40.989767076 +0000 UTC m=+867.715725111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-maas-api-tls") pod "maas-api-e2e-ait-65e77699-dc566c66-vvth8" (UID: "4669ed30-1ae0-487e-ac93-9f17e6cccf64") : secret "maas-api-e2e-ait-65e77699-serving-cert" not found
Sep 14 03:06:40.489992 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.489838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4669ed30-1ae0-487e-ac93-9f17e6cccf64-openshift-service-ca\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.489992 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.489879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tls-certs\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.490432 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.490411    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4669ed30-1ae0-487e-ac93-9f17e6cccf64-openshift-service-ca\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.491841 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.491824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-metrics-certs\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.499778 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.499755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfwsp\" (UniqueName: \"kubernetes.io/projected/4669ed30-1ae0-487e-ac93-9f17e6cccf64-kube-api-access-xfwsp\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.590805 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.590777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tls-certs\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.590919 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.590822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-home\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.590919 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.590866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-model-cache\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.590919 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.590905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86nbw\" (UniqueName: \"kubernetes.io/projected/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-kube-api-access-86nbw\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.591074 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.591012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tmp-dir\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.591137 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.591087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-dshm\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.591306 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.591285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-model-cache\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.591464 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.591443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tmp-dir\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.591535 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.591450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-home\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.593190 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.593170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-dshm\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.593248 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.593211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tls-certs\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.599523 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.599500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86nbw\" (UniqueName: \"kubernetes.io/projected/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-kube-api-access-86nbw\") pod \"autores-model-ab5750-kserve-b6cb4fdc5-5vtjn\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") " pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.759664 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.759634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:40.996903 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.996863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-maas-api-tls\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:40.999250 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:40.999220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-maas-api-tls\") pod \"maas-api-e2e-ait-65e77699-dc566c66-vvth8\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:41.040364 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:41.040341    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"]
Sep 14 03:06:41.042023 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:06:41.041991    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26e5b8f2_4b37_4f25_adaa_ef3bbfe8eb61.slice/crio-d2e7d9b13b5f5f20a0dd703771947d4d02e4f338416e99536c5d6bbb2ad2fd78 WatchSource:0}: Error finding container d2e7d9b13b5f5f20a0dd703771947d4d02e4f338416e99536c5d6bbb2ad2fd78: Status 404 returned error can't find the container with id d2e7d9b13b5f5f20a0dd703771947d4d02e4f338416e99536c5d6bbb2ad2fd78
Sep 14 03:06:41.236295 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:41.235993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:41.283094 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:41.283055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn" event={"ID":"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61","Type":"ContainerStarted","Data":"2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac"}
Sep 14 03:06:41.283240 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:41.283109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn" event={"ID":"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61","Type":"ContainerStarted","Data":"d2e7d9b13b5f5f20a0dd703771947d4d02e4f338416e99536c5d6bbb2ad2fd78"}
Sep 14 03:06:41.301094 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:41.300988    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn" podStartSLOduration=1.300966884 podStartE2EDuration="1.300966884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:06:41.299873883 +0000 UTC m=+868.025831925" watchObservedRunningTime="2026-09-14 03:06:41.300966884 +0000 UTC m=+868.026924939"
Sep 14 03:06:41.516414 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:41.516384    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"]
Sep 14 03:06:41.517067 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:06:41.517023    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4669ed30_1ae0_487e_ac93_9f17e6cccf64.slice/crio-e53382763ac2f431f1685ff77d82cde0694d1c519c8f85cdccc4a4f261104c42 WatchSource:0}: Error finding container e53382763ac2f431f1685ff77d82cde0694d1c519c8f85cdccc4a4f261104c42: Status 404 returned error can't find the container with id e53382763ac2f431f1685ff77d82cde0694d1c519c8f85cdccc4a4f261104c42
Sep 14 03:06:41.757689 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:41.757658    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e309690-02b8-4d64-8aa9-1002580cf5b6" path="/var/lib/kubelet/pods/9e309690-02b8-4d64-8aa9-1002580cf5b6/volumes"
Sep 14 03:06:42.293008 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:42.292927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8" event={"ID":"4669ed30-1ae0-487e-ac93-9f17e6cccf64","Type":"ContainerStarted","Data":"4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17"}
Sep 14 03:06:42.293008 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:42.292977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8" event={"ID":"4669ed30-1ae0-487e-ac93-9f17e6cccf64","Type":"ContainerStarted","Data":"e53382763ac2f431f1685ff77d82cde0694d1c519c8f85cdccc4a4f261104c42"}
Sep 14 03:06:42.293544 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:42.293199    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:42.318793 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:42.318739    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8" podStartSLOduration=1.8227925219999999 podStartE2EDuration="2.318723474s" totalImagesPullingTime="495.930952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:06:42.316201425 +0000 UTC m=+869.042159477" watchObservedRunningTime="2026-09-14 03:06:42.318723474 +0000 UTC m=+869.044681527"
Sep 14 03:06:48.304090 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:48.304064    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:06:50.159658 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:50.159626    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:06:50.760437 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:50.760397    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:50.760586 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:50.760458    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:50.772977 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:50.772952    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:06:51.357966 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:06:51.357937    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:07:00.407836 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:00.407799    2580 generic.go:356] "Generic (PLEG): container finished" podID="71dd5875-18de-4353-9917-44887c30e52e" containerID="d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39" exitCode=0
Sep 14 03:07:00.408283 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:00.407878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" event={"ID":"71dd5875-18de-4353-9917-44887c30e52e","Type":"ContainerDied","Data":"d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39"}
Sep 14 03:07:00.408336 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:00.408282    2580 scope.go:117] "RemoveContainer" containerID="d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39"
Sep 14 03:07:00.409538 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:00.409510    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:07:00.409601 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:00.409582    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:07:00.409639 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:00.409613    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:07:01.415746 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:01.415713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" event={"ID":"71dd5875-18de-4353-9917-44887c30e52e","Type":"ContainerStarted","Data":"96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4"}
Sep 14 03:07:01.932495 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:01.932458    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:07:01.932692 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:01.932669    2580 patch_prober.go:28] interesting pod/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 14 03:07:01.932739 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:01.932719    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 14 03:07:02.932749 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:02.932715    2580 patch_prober.go:28] interesting pod/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 14 03:07:02.933139 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:02.932766    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 14 03:07:03.932406 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:03.932371    2580 patch_prober.go:28] interesting pod/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 14 03:07:03.932585 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:03.932425    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 14 03:07:04.932355 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:04.932320    2580 patch_prober.go:28] interesting pod/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 14 03:07:04.932801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:04.932389    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 14 03:07:05.932229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:05.932191    2580 patch_prober.go:28] interesting pod/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 14 03:07:05.932403 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:05.932254    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 14 03:07:06.933141 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:06.933090    2580 patch_prober.go:28] interesting pod/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 14 03:07:06.933658 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:06.933174    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 14 03:07:07.459121 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.459089    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"]
Sep 14 03:07:07.462371 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.462349    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.464776 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.464752    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f002a1\"/\"openshift-service-ca.crt\""
Sep 14 03:07:07.464910 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.464890    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f002a1\"/\"kube-root-ca.crt\""
Sep 14 03:07:07.465393 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.465373    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f002a1\"/\"default-dockercfg-r7pjb\""
Sep 14 03:07:07.465547 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.465529    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f002a1\"/\"test-model-f002a1-kserve-self-signed-certs\""
Sep 14 03:07:07.477240 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.477215    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"]
Sep 14 03:07:07.545376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.545345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-model-cache\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.545551 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.545469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b1c738-5a1a-47b3-b285-7b862c45e499-tls-certs\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.545551 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.545531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-tmp-dir\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.545657 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.545566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57lsd\" (UniqueName: \"kubernetes.io/projected/53b1c738-5a1a-47b3-b285-7b862c45e499-kube-api-access-57lsd\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.545657 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.545586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-dshm\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.545734 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.545668    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-home\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.647160 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.647128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-home\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.647331 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.647166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-model-cache\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.647331 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.647225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b1c738-5a1a-47b3-b285-7b862c45e499-tls-certs\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.647331 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.647250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-tmp-dir\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.647331 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.647278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57lsd\" (UniqueName: \"kubernetes.io/projected/53b1c738-5a1a-47b3-b285-7b862c45e499-kube-api-access-57lsd\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.647331 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.647302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-dshm\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.647605 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.647541    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-home\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.647605 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.647576    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-model-cache\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.647687 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.647645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-tmp-dir\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.649587 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.649556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-dshm\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.649864 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.649845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b1c738-5a1a-47b3-b285-7b862c45e499-tls-certs\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.656452 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.656430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57lsd\" (UniqueName: \"kubernetes.io/projected/53b1c738-5a1a-47b3-b285-7b862c45e499-kube-api-access-57lsd\") pod \"test-model-f002a1-kserve-bc9954745-9qdrp\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") " pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.773729 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.773698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:07.936871 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.936839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:07:07.937322 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.937147    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:07:07.937904 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:07.937887    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:07:08.055492 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:08.055464    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"]
Sep 14 03:07:08.055607 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:07:08.055574    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53b1c738_5a1a_47b3_b285_7b862c45e499.slice/crio-b05527214d98408c3e073e6f953962b5a541aad5c8578f9637cf8524a6b7b484 WatchSource:0}: Error finding container b05527214d98408c3e073e6f953962b5a541aad5c8578f9637cf8524a6b7b484: Status 404 returned error can't find the container with id b05527214d98408c3e073e6f953962b5a541aad5c8578f9637cf8524a6b7b484
Sep 14 03:07:08.462210 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:08.462103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp" event={"ID":"53b1c738-5a1a-47b3-b285-7b862c45e499","Type":"ContainerStarted","Data":"e42cc379bdb0b20d3b8e41ff5369f51b9396d70f0d31c883e2b7eac8719190f4"}
Sep 14 03:07:08.462210 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:08.462150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp" event={"ID":"53b1c738-5a1a-47b3-b285-7b862c45e499","Type":"ContainerStarted","Data":"b05527214d98408c3e073e6f953962b5a541aad5c8578f9637cf8524a6b7b484"}
Sep 14 03:07:08.480445 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:08.480403    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp" podStartSLOduration=1.480388699 podStartE2EDuration="1.480388699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:07:08.479497284 +0000 UTC m=+895.205455336" watchObservedRunningTime="2026-09-14 03:07:08.480388699 +0000 UTC m=+895.206346745"
Sep 14 03:07:10.802759 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:10.802729    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"]
Sep 14 03:07:10.803152 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:10.802955    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8" podUID="4669ed30-1ae0-487e-ac93-9f17e6cccf64" containerName="maas-api" containerID="cri-o://4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17" gracePeriod=30
Sep 14 03:07:11.057296 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.057204    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:07:11.185112 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.185077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4669ed30-1ae0-487e-ac93-9f17e6cccf64-openshift-service-ca\") pod \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") "
Sep 14 03:07:11.185291 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.185206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-metrics-certs\") pod \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") "
Sep 14 03:07:11.185291 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.185249    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfwsp\" (UniqueName: \"kubernetes.io/projected/4669ed30-1ae0-487e-ac93-9f17e6cccf64-kube-api-access-xfwsp\") pod \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") "
Sep 14 03:07:11.185291 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.185282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-maas-api-tls\") pod \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\" (UID: \"4669ed30-1ae0-487e-ac93-9f17e6cccf64\") "
Sep 14 03:07:11.185451 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.185421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4669ed30-1ae0-487e-ac93-9f17e6cccf64-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4669ed30-1ae0-487e-ac93-9f17e6cccf64" (UID: "4669ed30-1ae0-487e-ac93-9f17e6cccf64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:07:11.185625 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.185608    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4669ed30-1ae0-487e-ac93-9f17e6cccf64-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:11.187419 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.187388    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4669ed30-1ae0-487e-ac93-9f17e6cccf64" (UID: "4669ed30-1ae0-487e-ac93-9f17e6cccf64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:07:11.187530 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.187457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4669ed30-1ae0-487e-ac93-9f17e6cccf64" (UID: "4669ed30-1ae0-487e-ac93-9f17e6cccf64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:07:11.187530 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.187476    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4669ed30-1ae0-487e-ac93-9f17e6cccf64-kube-api-access-xfwsp" (OuterVolumeSpecName: "kube-api-access-xfwsp") pod "4669ed30-1ae0-487e-ac93-9f17e6cccf64" (UID: "4669ed30-1ae0-487e-ac93-9f17e6cccf64"). InnerVolumeSpecName "kube-api-access-xfwsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:07:11.286574 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.286544    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:11.286574 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.286569    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfwsp\" (UniqueName: \"kubernetes.io/projected/4669ed30-1ae0-487e-ac93-9f17e6cccf64-kube-api-access-xfwsp\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:11.286574 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.286578    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4669ed30-1ae0-487e-ac93-9f17e6cccf64-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:11.480564 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.480478    2580 generic.go:356] "Generic (PLEG): container finished" podID="4669ed30-1ae0-487e-ac93-9f17e6cccf64" containerID="4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17" exitCode=0
Sep 14 03:07:11.480564 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.480522    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"
Sep 14 03:07:11.480753 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.480562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8" event={"ID":"4669ed30-1ae0-487e-ac93-9f17e6cccf64","Type":"ContainerDied","Data":"4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17"}
Sep 14 03:07:11.480753 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.480611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8" event={"ID":"4669ed30-1ae0-487e-ac93-9f17e6cccf64","Type":"ContainerDied","Data":"e53382763ac2f431f1685ff77d82cde0694d1c519c8f85cdccc4a4f261104c42"}
Sep 14 03:07:11.480753 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.480637    2580 scope.go:117] "RemoveContainer" containerID="4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17"
Sep 14 03:07:11.494165 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.493302    2580 scope.go:117] "RemoveContainer" containerID="4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17"
Sep 14 03:07:11.494165 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:11.493687    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17\": container with ID starting with 4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17 not found: ID does not exist" containerID="4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17"
Sep 14 03:07:11.494165 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.493721    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17"} err="failed to get container status \"4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17\": rpc error: code = NotFound desc = could not find container \"4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17\": container with ID starting with 4094a47bedd4b07af5fe0e94cd482c4ee1c15b4cb1e7db8a77637a2d96b37e17 not found: ID does not exist"
Sep 14 03:07:11.509918 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.509887    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"]
Sep 14 03:07:11.516810 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.516787    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65e77699-dc566c66-vvth8"]
Sep 14 03:07:11.758111 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:11.758075    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4669ed30-1ae0-487e-ac93-9f17e6cccf64" path="/var/lib/kubelet/pods/4669ed30-1ae0-487e-ac93-9f17e6cccf64/volumes"
Sep 14 03:07:12.159572 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.159491    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"]
Sep 14 03:07:12.159993 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.159679    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy" containerID="cri-o://96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4" gracePeriod=30
Sep 14 03:07:12.280189 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.280158    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"]
Sep 14 03:07:12.280737 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.280722    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4669ed30-1ae0-487e-ac93-9f17e6cccf64" containerName="maas-api"
Sep 14 03:07:12.280781 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.280742    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4669ed30-1ae0-487e-ac93-9f17e6cccf64" containerName="maas-api"
Sep 14 03:07:12.280856 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.280845    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4669ed30-1ae0-487e-ac93-9f17e6cccf64" containerName="maas-api"
Sep 14 03:07:12.284714 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.284691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"
Sep 14 03:07:12.287008 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.286980    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:07:12.287392 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.287378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sv5sd\""
Sep 14 03:07:12.294673 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.294647    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"]
Sep 14 03:07:12.398511 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.398481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8390acfd-a6e3-4cc2-a11d-f440545cda08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ab5750-kk5nt\" (UID: \"8390acfd-a6e3-4cc2-a11d-f440545cda08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"
Sep 14 03:07:12.499592 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.499506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8390acfd-a6e3-4cc2-a11d-f440545cda08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ab5750-kk5nt\" (UID: \"8390acfd-a6e3-4cc2-a11d-f440545cda08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"
Sep 14 03:07:12.500269 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.500236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8390acfd-a6e3-4cc2-a11d-f440545cda08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ab5750-kk5nt\" (UID: \"8390acfd-a6e3-4cc2-a11d-f440545cda08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"
Sep 14 03:07:12.597317 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.597278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"
Sep 14 03:07:12.879825 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:12.879799    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"]
Sep 14 03:07:12.882364 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:07:12.882333    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f WatchSource:0}: Error finding container 88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f: Status 404 returned error can't find the container with id 88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f
Sep 14 03:07:13.496558 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:13.496457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt" event={"ID":"8390acfd-a6e3-4cc2-a11d-f440545cda08","Type":"ContainerStarted","Data":"0e27c5dc11c0034b5df7abf7247961150d5aa1cee199420247711e7f70f7fca5"}
Sep 14 03:07:13.496558 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:13.496549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt" event={"ID":"8390acfd-a6e3-4cc2-a11d-f440545cda08","Type":"ContainerStarted","Data":"88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f"}
Sep 14 03:07:13.518310 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:13.518264    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt" podStartSLOduration=1.5182488950000002 podStartE2EDuration="1.518248895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:07:13.515782891 +0000 UTC m=+900.241740943" watchObservedRunningTime="2026-09-14 03:07:13.518248895 +0000 UTC m=+900.244206946"
Sep 14 03:07:13.755762 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:13.755589    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:07:13.756875 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:13.756626    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:07:14.504237 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:14.504206    2580 generic.go:356] "Generic (PLEG): container finished" podID="8390acfd-a6e3-4cc2-a11d-f440545cda08" containerID="0e27c5dc11c0034b5df7abf7247961150d5aa1cee199420247711e7f70f7fca5" exitCode=0
Sep 14 03:07:14.504628 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:14.504293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt" event={"ID":"8390acfd-a6e3-4cc2-a11d-f440545cda08","Type":"ContainerDied","Data":"0e27c5dc11c0034b5df7abf7247961150d5aa1cee199420247711e7f70f7fca5"}
Sep 14 03:07:15.636342 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:15.636319    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"
Sep 14 03:07:15.732123 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:15.732095    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8390acfd-a6e3-4cc2-a11d-f440545cda08-maas-api-service-ca\") pod \"8390acfd-a6e3-4cc2-a11d-f440545cda08\" (UID: \"8390acfd-a6e3-4cc2-a11d-f440545cda08\") "
Sep 14 03:07:15.732414 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:15.732394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8390acfd-a6e3-4cc2-a11d-f440545cda08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8390acfd-a6e3-4cc2-a11d-f440545cda08" (UID: "8390acfd-a6e3-4cc2-a11d-f440545cda08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:07:15.834383 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:15.834347    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8390acfd-a6e3-4cc2-a11d-f440545cda08-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:16.517867 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:16.517834    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"
Sep 14 03:07:16.518064 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:16.517874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt" event={"ID":"8390acfd-a6e3-4cc2-a11d-f440545cda08","Type":"ContainerDied","Data":"88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f"}
Sep 14 03:07:16.518064 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:16.517948    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f"
Sep 14 03:07:17.278525 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.278501    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:07:17.422894 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.422840    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:07:17.447332 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447309    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-credential-socket\") pod \"71dd5875-18de-4353-9917-44887c30e52e\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") "
Sep 14 03:07:17.447421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447344    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-istio-token\") pod \"71dd5875-18de-4353-9917-44887c30e52e\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") "
Sep 14 03:07:17.447421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-data\") pod \"71dd5875-18de-4353-9917-44887c30e52e\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") "
Sep 14 03:07:17.447421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447391    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dd5875-18de-4353-9917-44887c30e52e-istio-podinfo\") pod \"71dd5875-18de-4353-9917-44887c30e52e\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") "
Sep 14 03:07:17.447421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-envoy\") pod \"71dd5875-18de-4353-9917-44887c30e52e\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") "
Sep 14 03:07:17.447601 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447492    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-socket\") pod \"71dd5875-18de-4353-9917-44887c30e52e\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") "
Sep 14 03:07:17.447601 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447540    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dd5875-18de-4353-9917-44887c30e52e-istiod-ca-cert\") pod \"71dd5875-18de-4353-9917-44887c30e52e\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") "
Sep 14 03:07:17.447601 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447565    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71dd5875-18de-4353-9917-44887c30e52e" (UID: "71dd5875-18de-4353-9917-44887c30e52e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:07:17.447601 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447574    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-certs\") pod \"71dd5875-18de-4353-9917-44887c30e52e\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") "
Sep 14 03:07:17.447808 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddn9p\" (UniqueName: \"kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-kube-api-access-ddn9p\") pod \"71dd5875-18de-4353-9917-44887c30e52e\" (UID: \"71dd5875-18de-4353-9917-44887c30e52e\") "
Sep 14 03:07:17.447808 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-data" (OuterVolumeSpecName: "istio-data") pod "71dd5875-18de-4353-9917-44887c30e52e" (UID: "71dd5875-18de-4353-9917-44887c30e52e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:07:17.447808 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447777    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71dd5875-18de-4353-9917-44887c30e52e" (UID: "71dd5875-18de-4353-9917-44887c30e52e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:07:17.448299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447843    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71dd5875-18de-4353-9917-44887c30e52e" (UID: "71dd5875-18de-4353-9917-44887c30e52e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:07:17.448299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.447977    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71dd5875-18de-4353-9917-44887c30e52e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71dd5875-18de-4353-9917-44887c30e52e" (UID: "71dd5875-18de-4353-9917-44887c30e52e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:07:17.448299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.448125    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:17.448299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.448146    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dd5875-18de-4353-9917-44887c30e52e-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:17.448299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.448173    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:17.448299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.448186    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:17.448299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.448198    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:17.450220 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.450192    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71dd5875-18de-4353-9917-44887c30e52e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71dd5875-18de-4353-9917-44887c30e52e" (UID: "71dd5875-18de-4353-9917-44887c30e52e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 03:07:17.450363 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.450303    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-istio-token" (OuterVolumeSpecName: "istio-token") pod "71dd5875-18de-4353-9917-44887c30e52e" (UID: "71dd5875-18de-4353-9917-44887c30e52e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:07:17.450480 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.450455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71dd5875-18de-4353-9917-44887c30e52e" (UID: "71dd5875-18de-4353-9917-44887c30e52e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:07:17.450628 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.450532    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-kube-api-access-ddn9p" (OuterVolumeSpecName: "kube-api-access-ddn9p") pod "71dd5875-18de-4353-9917-44887c30e52e" (UID: "71dd5875-18de-4353-9917-44887c30e52e"). InnerVolumeSpecName "kube-api-access-ddn9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:07:17.525517 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.525489    2580 generic.go:356] "Generic (PLEG): container finished" podID="71dd5875-18de-4353-9917-44887c30e52e" containerID="96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4" exitCode=0
Sep 14 03:07:17.525660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.525532    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"
Sep 14 03:07:17.525660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.525567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" event={"ID":"71dd5875-18de-4353-9917-44887c30e52e","Type":"ContainerDied","Data":"96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4"}
Sep 14 03:07:17.525660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.525623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm" event={"ID":"71dd5875-18de-4353-9917-44887c30e52e","Type":"ContainerDied","Data":"693d8c3cfee81c4478c4a56d0165516f025ee85f477bf15cb90e6a6fa170870b"}
Sep 14 03:07:17.525660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.525654    2580 scope.go:117] "RemoveContainer" containerID="96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4"
Sep 14 03:07:17.535191 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.535172    2580 scope.go:117] "RemoveContainer" containerID="d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39"
Sep 14 03:07:17.543775 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.543749    2580 scope.go:117] "RemoveContainer" containerID="96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4"
Sep 14 03:07:17.544091 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:17.544067    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4\": container with ID starting with 96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4 not found: ID does not exist" containerID="96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4"
Sep 14 03:07:17.544158 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.544101    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4"} err="failed to get container status \"96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4\": rpc error: code = NotFound desc = could not find container \"96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4\": container with ID starting with 96fe0a2c57949464babfa9108603b2d2b9a8e953e30df559277b50a73c9ebcc4 not found: ID does not exist"
Sep 14 03:07:17.544158 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.544126    2580 scope.go:117] "RemoveContainer" containerID="d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39"
Sep 14 03:07:17.544392 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:17.544377    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39\": container with ID starting with d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39 not found: ID does not exist" containerID="d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39"
Sep 14 03:07:17.544438 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.544396    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39"} err="failed to get container status \"d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39\": rpc error: code = NotFound desc = could not find container \"d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39\": container with ID starting with d340590a7dc132b168d1f999566129eeaf11695fe18f5fa48741556e4b9bfc39 not found: ID does not exist"
Sep 14 03:07:17.548916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.548898    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddn9p\" (UniqueName: \"kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-kube-api-access-ddn9p\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:17.548974 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.548918    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dd5875-18de-4353-9917-44887c30e52e-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:17.548974 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.548928    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dd5875-18de-4353-9917-44887c30e52e-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:17.548974 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.548936    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dd5875-18de-4353-9917-44887c30e52e-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:17.553964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.553946    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"]
Sep 14 03:07:17.558486 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.558466    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-65e77699-openshift-default-9cd4966fb-d2lsm"]
Sep 14 03:07:17.757737 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.757712    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71dd5875-18de-4353-9917-44887c30e52e" path="/var/lib/kubelet/pods/71dd5875-18de-4353-9917-44887c30e52e/volumes"
Sep 14 03:07:17.774093 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.774070    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:17.774356 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.774330    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:17.787134 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:17.787118    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:18.428909 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.428875    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"]
Sep 14 03:07:18.429314 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.429300    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8390acfd-a6e3-4cc2-a11d-f440545cda08" containerName="revoke-keys"
Sep 14 03:07:18.429357 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.429317    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8390acfd-a6e3-4cc2-a11d-f440545cda08" containerName="revoke-keys"
Sep 14 03:07:18.429357 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.429331    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy"
Sep 14 03:07:18.429357 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.429337    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy"
Sep 14 03:07:18.429357 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.429353    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy"
Sep 14 03:07:18.429481 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.429360    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy"
Sep 14 03:07:18.429481 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.429426    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy"
Sep 14 03:07:18.429481 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.429437    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8390acfd-a6e3-4cc2-a11d-f440545cda08" containerName="revoke-keys"
Sep 14 03:07:18.432609 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.432591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:18.436445 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.436424    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fb76f4c2-serving-cert\""
Sep 14 03:07:18.436744 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.436717    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jf78h\""
Sep 14 03:07:18.437217 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.437188    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 03:07:18.437313 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.437196    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:07:18.437313 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.437283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 03:07:18.444703 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.444678    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"]
Sep 14 03:07:18.458740 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.458713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-openshift-service-ca\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:18.458882 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.458764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-metrics-certs\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:18.458952 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.458905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-maas-api-tls\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:18.459014 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.458948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d878g\" (UniqueName: \"kubernetes.io/projected/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-kube-api-access-d878g\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:18.564410 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.562365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-openshift-service-ca\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:18.564410 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.562433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-metrics-certs\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:18.564410 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.562583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-maas-api-tls\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:18.564410 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.562630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d878g\" (UniqueName: \"kubernetes.io/projected/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-kube-api-access-d878g\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:18.564410 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.563921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-openshift-service-ca\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:18.565456 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:18.565433    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-serving-cert: secret "maas-api-e2e-derive-fb76f4c2-serving-cert" not found
Sep 14 03:07:18.565588 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:18.565523    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-maas-api-tls podName:e9da60f3-8705-49d0-b9f6-92fffcd99f9d nodeName:}" failed. No retries permitted until 2026-09-14 03:07:19.065502643 +0000 UTC m=+905.791460673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-maas-api-tls") pod "maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn" (UID: "e9da60f3-8705-49d0-b9f6-92fffcd99f9d") : secret "maas-api-e2e-derive-fb76f4c2-serving-cert" not found
Sep 14 03:07:18.571008 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.570978    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:07:18.573146 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.573110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-metrics-certs\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:18.574169 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:18.574149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d878g\" (UniqueName: \"kubernetes.io/projected/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-kube-api-access-d878g\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:19.068721 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:19.068687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-maas-api-tls\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:19.071181 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:19.071148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-maas-api-tls\") pod \"maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:19.346785 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:19.346703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:19.624427 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:19.624352    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"]
Sep 14 03:07:19.627365 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:07:19.627339    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9da60f3_8705_49d0_b9f6_92fffcd99f9d.slice/crio-4469a9fc298ccc6f5b2ba5612e589d6c3cc34a5b1b09be4c59f48306a05880c6 WatchSource:0}: Error finding container 4469a9fc298ccc6f5b2ba5612e589d6c3cc34a5b1b09be4c59f48306a05880c6: Status 404 returned error can't find the container with id 4469a9fc298ccc6f5b2ba5612e589d6c3cc34a5b1b09be4c59f48306a05880c6
Sep 14 03:07:20.549497 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:20.549454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn" event={"ID":"e9da60f3-8705-49d0-b9f6-92fffcd99f9d","Type":"ContainerStarted","Data":"7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19"}
Sep 14 03:07:20.549699 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:20.549533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:20.549699 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:20.549558    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn" event={"ID":"e9da60f3-8705-49d0-b9f6-92fffcd99f9d","Type":"ContainerStarted","Data":"4469a9fc298ccc6f5b2ba5612e589d6c3cc34a5b1b09be4c59f48306a05880c6"}
Sep 14 03:07:20.568510 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:20.568461    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn" podStartSLOduration=2.005178025 podStartE2EDuration="2.568446101s" totalImagesPullingTime="563.268076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:07:20.567194864 +0000 UTC m=+907.293152920" watchObservedRunningTime="2026-09-14 03:07:20.568446101 +0000 UTC m=+907.294404153"
Sep 14 03:07:20.672485 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:20.672452    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f\": RecentStats: unable to find data in memory cache]"
Sep 14 03:07:23.295787 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:23.295751    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"]
Sep 14 03:07:23.298260 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:23.298239    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ab5750-kk5nt"]
Sep 14 03:07:23.758262 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:23.758227    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8390acfd-a6e3-4cc2-a11d-f440545cda08" path="/var/lib/kubelet/pods/8390acfd-a6e3-4cc2-a11d-f440545cda08/volumes"
Sep 14 03:07:26.053858 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:26.053820    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f\": RecentStats: unable to find data in memory cache]"
Sep 14 03:07:26.561312 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:26.561287    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:28.657302 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:28.657267    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"]
Sep 14 03:07:31.772512 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:31.772484    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f\": RecentStats: unable to find data in memory cache]"
Sep 14 03:07:31.772904 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:31.772482    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f\": RecentStats: unable to find data in memory cache]"
Sep 14 03:07:35.664635 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:35.664599    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f\": RecentStats: unable to find data in memory cache]"
Sep 14 03:07:36.064511 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:36.064475    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f\": RecentStats: unable to find data in memory cache]"
Sep 14 03:07:39.968257 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:39.968224    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"]
Sep 14 03:07:39.968688 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:39.968413    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn" podUID="e9da60f3-8705-49d0-b9f6-92fffcd99f9d" containerName="maas-api" containerID="cri-o://7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19" gracePeriod=30
Sep 14 03:07:40.212207 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.212183    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:40.275624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.275596    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-maas-api-tls\") pod \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") "
Sep 14 03:07:40.275760 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.275673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-openshift-service-ca\") pod \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") "
Sep 14 03:07:40.275760 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.275708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d878g\" (UniqueName: \"kubernetes.io/projected/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-kube-api-access-d878g\") pod \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") "
Sep 14 03:07:40.275848 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.275787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-metrics-certs\") pod \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\" (UID: \"e9da60f3-8705-49d0-b9f6-92fffcd99f9d\") "
Sep 14 03:07:40.276102 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.276077    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e9da60f3-8705-49d0-b9f6-92fffcd99f9d" (UID: "e9da60f3-8705-49d0-b9f6-92fffcd99f9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:07:40.276207 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.276163    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:40.277693 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.277660    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9da60f3-8705-49d0-b9f6-92fffcd99f9d" (UID: "e9da60f3-8705-49d0-b9f6-92fffcd99f9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:07:40.277801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.277742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-kube-api-access-d878g" (OuterVolumeSpecName: "kube-api-access-d878g") pod "e9da60f3-8705-49d0-b9f6-92fffcd99f9d" (UID: "e9da60f3-8705-49d0-b9f6-92fffcd99f9d"). InnerVolumeSpecName "kube-api-access-d878g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:07:40.277885 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.277865    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e9da60f3-8705-49d0-b9f6-92fffcd99f9d" (UID: "e9da60f3-8705-49d0-b9f6-92fffcd99f9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:07:40.377537 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.377506    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:40.377537 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.377530    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:40.377537 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.377539    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d878g\" (UniqueName: \"kubernetes.io/projected/e9da60f3-8705-49d0-b9f6-92fffcd99f9d-kube-api-access-d878g\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:40.665758 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.665670    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9da60f3-8705-49d0-b9f6-92fffcd99f9d" containerID="7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19" exitCode=0
Sep 14 03:07:40.665758 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.665718    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"
Sep 14 03:07:40.665758 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.665751    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn" event={"ID":"e9da60f3-8705-49d0-b9f6-92fffcd99f9d","Type":"ContainerDied","Data":"7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19"}
Sep 14 03:07:40.666004 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.665801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn" event={"ID":"e9da60f3-8705-49d0-b9f6-92fffcd99f9d","Type":"ContainerDied","Data":"4469a9fc298ccc6f5b2ba5612e589d6c3cc34a5b1b09be4c59f48306a05880c6"}
Sep 14 03:07:40.666004 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.665823    2580 scope.go:117] "RemoveContainer" containerID="7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19"
Sep 14 03:07:40.683924 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.683899    2580 scope.go:117] "RemoveContainer" containerID="7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19"
Sep 14 03:07:40.684400 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:40.684379    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19\": container with ID starting with 7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19 not found: ID does not exist" containerID="7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19"
Sep 14 03:07:40.684479 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.684407    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19"} err="failed to get container status \"7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19\": rpc error: code = NotFound desc = could not find container \"7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19\": container with ID starting with 7e0b2ef23ba6dd85ed6f2f6a0f2c26f4451f2d56efb427ea6936f0e41545af19 not found: ID does not exist"
Sep 14 03:07:40.691156 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.691131    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"]
Sep 14 03:07:40.697612 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:40.697582    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb76f4c2-cc68c57b5-bcwpn"]
Sep 14 03:07:41.756864 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:41.756831    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9da60f3-8705-49d0-b9f6-92fffcd99f9d" path="/var/lib/kubelet/pods/e9da60f3-8705-49d0-b9f6-92fffcd99f9d/volumes"
Sep 14 03:07:43.676580 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:43.676544    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn" podUID="26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" containerName="main" containerID="cri-o://2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac" gracePeriod=45
Sep 14 03:07:43.920975 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:43.920953    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:07:43.961620 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:43.961541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:07:44.013246 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tmp-dir\") pod \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") "
Sep 14 03:07:44.013412 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013255    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-home\") pod \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") "
Sep 14 03:07:44.013412 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013292    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-model-cache\") pod \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") "
Sep 14 03:07:44.013412 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86nbw\" (UniqueName: \"kubernetes.io/projected/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-kube-api-access-86nbw\") pod \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") "
Sep 14 03:07:44.013412 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-dshm\") pod \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") "
Sep 14 03:07:44.013412 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tls-certs\") pod \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\" (UID: \"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61\") "
Sep 14 03:07:44.013667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" (UID: "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:07:44.013667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-home" (OuterVolumeSpecName: "home") pod "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" (UID: "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:07:44.013667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013549    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-model-cache" (OuterVolumeSpecName: "model-cache") pod "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" (UID: "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:07:44.013822 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013767    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:44.013822 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013784    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:44.013822 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.013797    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:44.015558 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.015523    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-dshm" (OuterVolumeSpecName: "dshm") pod "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" (UID: "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:07:44.015558 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.015534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-kube-api-access-86nbw" (OuterVolumeSpecName: "kube-api-access-86nbw") pod "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" (UID: "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61"). InnerVolumeSpecName "kube-api-access-86nbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:07:44.015558 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.015534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" (UID: "26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:07:44.115040 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.114999    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86nbw\" (UniqueName: \"kubernetes.io/projected/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-kube-api-access-86nbw\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:44.115180 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.115024    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:44.115180 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.115058    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:07:44.692841 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.692802    2580 generic.go:356] "Generic (PLEG): container finished" podID="26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" containerID="2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac" exitCode=0
Sep 14 03:07:44.692841 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.692840    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"
Sep 14 03:07:44.693343 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.692875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn" event={"ID":"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61","Type":"ContainerDied","Data":"2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac"}
Sep 14 03:07:44.693343 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.692919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn" event={"ID":"26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61","Type":"ContainerDied","Data":"d2e7d9b13b5f5f20a0dd703771947d4d02e4f338416e99536c5d6bbb2ad2fd78"}
Sep 14 03:07:44.693343 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.692939    2580 scope.go:117] "RemoveContainer" containerID="2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac"
Sep 14 03:07:44.703061 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.703025    2580 scope.go:117] "RemoveContainer" containerID="2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac"
Sep 14 03:07:44.703327 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:44.703309    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac\": container with ID starting with 2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac not found: ID does not exist" containerID="2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac"
Sep 14 03:07:44.703372 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.703332    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac"} err="failed to get container status \"2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac\": rpc error: code = NotFound desc = could not find container \"2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac\": container with ID starting with 2feafdc9caeebd66c3d0345aea073c29f0fd41ff1fc5921311cdeb475c408dac not found: ID does not exist"
Sep 14 03:07:44.714560 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.714534    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"]
Sep 14 03:07:44.720439 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:44.720414    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-ab5750/autores-model-ab5750-kserve-b6cb4fdc5-5vtjn"]
Sep 14 03:07:45.756260 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:07:45.756228    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" path="/var/lib/kubelet/pods/26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61/volumes"
Sep 14 03:07:46.112115 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:46.112021    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f\": RecentStats: unable to find data in memory cache]"
Sep 14 03:07:50.659825 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:50.659788    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f\": RecentStats: unable to find data in memory cache]"
Sep 14 03:07:56.158509 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:07:56.158478    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f\": RecentStats: unable to find data in memory cache]"
Sep 14 03:08:05.718614 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:08:05.718583    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f\": RecentStats: unable to find data in memory cache]"
Sep 14 03:08:06.003434 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:06.003363    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"]
Sep 14 03:08:06.003611 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:06.003568    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy" containerID="cri-o://3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae" gracePeriod=30
Sep 14 03:08:06.169243 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:08:06.169208    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390acfd_a6e3_4cc2_a11d_f440545cda08.slice/crio-88f5e34be38885983749efc969e576c465ed4647637fdc6ea24e0e26b5c7f70f\": RecentStats: unable to find data in memory cache]"
Sep 14 03:08:07.872067 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:07.872025    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:08:11.258880 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.258856    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:08:11.359804 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.359743    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-credential-socket\") pod \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") "
Sep 14 03:08:11.359804 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.359781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-data\") pod \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") "
Sep 14 03:08:11.360058 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.359813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-socket\") pod \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") "
Sep 14 03:08:11.360058 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.359868    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-envoy\") pod \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") "
Sep 14 03:08:11.360058 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.359920    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dqht\" (UniqueName: \"kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-kube-api-access-6dqht\") pod \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") "
Sep 14 03:08:11.360058 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.359965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istiod-ca-cert\") pod \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") "
Sep 14 03:08:11.360058 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.359990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-token\") pod \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") "
Sep 14 03:08:11.360348 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.360077    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c84f7dd1-fb46-4476-badb-19eb0a7997f5" (UID: "c84f7dd1-fb46-4476-badb-19eb0a7997f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:08:11.360348 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.360099    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-podinfo\") pod \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") "
Sep 14 03:08:11.360348 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.360097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "c84f7dd1-fb46-4476-badb-19eb0a7997f5" (UID: "c84f7dd1-fb46-4476-badb-19eb0a7997f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:08:11.360348 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.360153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-certs\") pod \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\" (UID: \"c84f7dd1-fb46-4476-badb-19eb0a7997f5\") "
Sep 14 03:08:11.360348 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.360196    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c84f7dd1-fb46-4476-badb-19eb0a7997f5" (UID: "c84f7dd1-fb46-4476-badb-19eb0a7997f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:08:11.360643 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.360442    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:08:11.360643 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.360448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c84f7dd1-fb46-4476-badb-19eb0a7997f5" (UID: "c84f7dd1-fb46-4476-badb-19eb0a7997f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:08:11.360643 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.360462    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:08:11.360643 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.360477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c84f7dd1-fb46-4476-badb-19eb0a7997f5" (UID: "c84f7dd1-fb46-4476-badb-19eb0a7997f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:08:11.360643 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.360506    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:08:11.362229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.362200    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "c84f7dd1-fb46-4476-badb-19eb0a7997f5" (UID: "c84f7dd1-fb46-4476-badb-19eb0a7997f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:08:11.362342 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.362243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-kube-api-access-6dqht" (OuterVolumeSpecName: "kube-api-access-6dqht") pod "c84f7dd1-fb46-4476-badb-19eb0a7997f5" (UID: "c84f7dd1-fb46-4476-badb-19eb0a7997f5"). InnerVolumeSpecName "kube-api-access-6dqht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:08:11.362342 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.362284    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c84f7dd1-fb46-4476-badb-19eb0a7997f5" (UID: "c84f7dd1-fb46-4476-badb-19eb0a7997f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 03:08:11.362342 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.362280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c84f7dd1-fb46-4476-badb-19eb0a7997f5" (UID: "c84f7dd1-fb46-4476-badb-19eb0a7997f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:08:11.461908 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.461867    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:08:11.461908 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.461903    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:08:11.461908 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.461912    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:08:11.461908 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.461920    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dqht\" (UniqueName: \"kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-kube-api-access-6dqht\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:08:11.462194 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.461930    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:08:11.462194 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.461938    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84f7dd1-fb46-4476-badb-19eb0a7997f5-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:08:11.851711 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.851675    2580 generic.go:356] "Generic (PLEG): container finished" podID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerID="3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae" exitCode=0
Sep 14 03:08:11.851864 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.851727    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"
Sep 14 03:08:11.851864 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.851759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" event={"ID":"c84f7dd1-fb46-4476-badb-19eb0a7997f5","Type":"ContainerDied","Data":"3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae"}
Sep 14 03:08:11.851864 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.851814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps" event={"ID":"c84f7dd1-fb46-4476-badb-19eb0a7997f5","Type":"ContainerDied","Data":"de0a95da74c1950f6a1029af4a803645c09a9c9dbf031d0e52afe3fcefc325b8"}
Sep 14 03:08:11.851864 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.851837    2580 scope.go:117] "RemoveContainer" containerID="3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae"
Sep 14 03:08:11.863830 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.863813    2580 scope.go:117] "RemoveContainer" containerID="22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0"
Sep 14 03:08:11.872166 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.872150    2580 scope.go:117] "RemoveContainer" containerID="3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae"
Sep 14 03:08:11.872464 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:08:11.872444    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae\": container with ID starting with 3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae not found: ID does not exist" containerID="3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae"
Sep 14 03:08:11.872521 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.872471    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae"} err="failed to get container status \"3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae\": rpc error: code = NotFound desc = could not find container \"3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae\": container with ID starting with 3e7a330f1f1d61b641580074ea76cf672f645a5cd193787625465cc31277b7ae not found: ID does not exist"
Sep 14 03:08:11.872521 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.872487    2580 scope.go:117] "RemoveContainer" containerID="22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0"
Sep 14 03:08:11.872750 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:08:11.872726    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0\": container with ID starting with 22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0 not found: ID does not exist" containerID="22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0"
Sep 14 03:08:11.872822 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.872759    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0"} err="failed to get container status \"22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0\": rpc error: code = NotFound desc = could not find container \"22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0\": container with ID starting with 22812fe94538457863b5dffc05788e9661fd843989a421ac5ed5ea74ddb777b0 not found: ID does not exist"
Sep 14 03:08:11.876746 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.876727    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"]
Sep 14 03:08:11.881552 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:11.881533    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-ab5750-openshift-default-55b755ff66-fr5ps"]
Sep 14 03:08:12.146768 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.146698    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"]
Sep 14 03:08:12.147074 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147062    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9da60f3-8705-49d0-b9f6-92fffcd99f9d" containerName="maas-api"
Sep 14 03:08:12.147126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147076    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9da60f3-8705-49d0-b9f6-92fffcd99f9d" containerName="maas-api"
Sep 14 03:08:12.147126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147092    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy"
Sep 14 03:08:12.147126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147097    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy"
Sep 14 03:08:12.147126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147109    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" containerName="main"
Sep 14 03:08:12.147126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147114    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" containerName="main"
Sep 14 03:08:12.147126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147122    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy"
Sep 14 03:08:12.147126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147127    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy"
Sep 14 03:08:12.147334 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147188    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy"
Sep 14 03:08:12.147334 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147195    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9da60f3-8705-49d0-b9f6-92fffcd99f9d" containerName="maas-api"
Sep 14 03:08:12.147334 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147202    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71dd5875-18de-4353-9917-44887c30e52e" containerName="istio-proxy"
Sep 14 03:08:12.147334 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.147209    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e5b8f2-4b37-4f25-adaa-ef3bbfe8eb61" containerName="main"
Sep 14 03:08:12.151526 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.151508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.153571 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.153551    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a752f\"/\"premium-w0-3a752f-kserve-self-signed-certs\""
Sep 14 03:08:12.159842 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.159819    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"]
Sep 14 03:08:12.267686 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.267655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-model-cache\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.268126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.267693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnmp4\" (UniqueName: \"kubernetes.io/projected/07863119-ad92-4bf2-bb9e-8b851262fbe3-kube-api-access-cnmp4\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.268126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.267723    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-home\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.268126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.267795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-dshm\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.268126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.267851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07863119-ad92-4bf2-bb9e-8b851262fbe3-tls-certs\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.268126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.267903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-tmp-dir\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.369053 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.369003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-tmp-dir\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.369206 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.369112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-model-cache\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.369206 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.369134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnmp4\" (UniqueName: \"kubernetes.io/projected/07863119-ad92-4bf2-bb9e-8b851262fbe3-kube-api-access-cnmp4\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.369206 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.369166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-home\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.369206 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.369190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-dshm\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.369417 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.369309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07863119-ad92-4bf2-bb9e-8b851262fbe3-tls-certs\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.369417 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.369375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-tmp-dir\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.369533 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.369427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-model-cache\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.369590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.369554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-home\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.371528 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.371504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-dshm\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.371682 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.371665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07863119-ad92-4bf2-bb9e-8b851262fbe3-tls-certs\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.377702 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.377682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnmp4\" (UniqueName: \"kubernetes.io/projected/07863119-ad92-4bf2-bb9e-8b851262fbe3-kube-api-access-cnmp4\") pod \"premium-w0-3a752f-kserve-769b69d8c6-htrs5\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") " pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.461759 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.461676    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:12.738660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.738578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"]
Sep 14 03:08:12.741585 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:08:12.741555    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07863119_ad92_4bf2_bb9e_8b851262fbe3.slice/crio-36be7669675c4ac9ca3c28cfc963ea94d266889b439a0740cbd49c09a327de66 WatchSource:0}: Error finding container 36be7669675c4ac9ca3c28cfc963ea94d266889b439a0740cbd49c09a327de66: Status 404 returned error can't find the container with id 36be7669675c4ac9ca3c28cfc963ea94d266889b439a0740cbd49c09a327de66
Sep 14 03:08:12.861459 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.861430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5" event={"ID":"07863119-ad92-4bf2-bb9e-8b851262fbe3","Type":"ContainerStarted","Data":"a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382"}
Sep 14 03:08:12.861459 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.861470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5" event={"ID":"07863119-ad92-4bf2-bb9e-8b851262fbe3","Type":"ContainerStarted","Data":"36be7669675c4ac9ca3c28cfc963ea94d266889b439a0740cbd49c09a327de66"}
Sep 14 03:08:12.882243 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:12.882195    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5" podStartSLOduration=0.882179099 podStartE2EDuration="882.179099ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:08:12.880368574 +0000 UTC m=+959.606326627" watchObservedRunningTime="2026-09-14 03:08:12.882179099 +0000 UTC m=+959.608137151"
Sep 14 03:08:13.757272 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:13.757238    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" path="/var/lib/kubelet/pods/c84f7dd1-fb46-4476-badb-19eb0a7997f5/volumes"
Sep 14 03:08:14.123021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.122951    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"]
Sep 14 03:08:14.123419 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.123406    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c84f7dd1-fb46-4476-badb-19eb0a7997f5" containerName="istio-proxy"
Sep 14 03:08:14.127914 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.127898    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:08:14.131378 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.131354    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-8364a0-dockercfg-5j9rf\""
Sep 14 03:08:14.131716 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.131699    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-8364a0\""
Sep 14 03:08:14.144652 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.144628    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"]
Sep 14 03:08:14.187266 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.187210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd\" (UID: \"9ca51ec5-2f3a-4372-92a7-908e2ba739dc\") " pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:08:14.187415 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.187285    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqkp6\" (UniqueName: \"kubernetes.io/projected/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-kube-api-access-gqkp6\") pod \"payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd\" (UID: \"9ca51ec5-2f3a-4372-92a7-908e2ba739dc\") " pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:08:14.288281 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.288249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqkp6\" (UniqueName: \"kubernetes.io/projected/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-kube-api-access-gqkp6\") pod \"payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd\" (UID: \"9ca51ec5-2f3a-4372-92a7-908e2ba739dc\") " pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:08:14.288475 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.288373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd\" (UID: \"9ca51ec5-2f3a-4372-92a7-908e2ba739dc\") " pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:08:14.289014 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.288995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd\" (UID: \"9ca51ec5-2f3a-4372-92a7-908e2ba739dc\") " pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:08:14.302025 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.301995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqkp6\" (UniqueName: \"kubernetes.io/projected/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-kube-api-access-gqkp6\") pod \"payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd\" (UID: \"9ca51ec5-2f3a-4372-92a7-908e2ba739dc\") " pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:08:14.437523 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.437454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:08:14.710718 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.710689    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"]
Sep 14 03:08:14.711156 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:08:14.711129    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca51ec5_2f3a_4372_92a7_908e2ba739dc.slice/crio-8c4584cc390448493c2fa5c4e4dbe0203f8c22d4e68a54740c93a7dba966fdaa WatchSource:0}: Error finding container 8c4584cc390448493c2fa5c4e4dbe0203f8c22d4e68a54740c93a7dba966fdaa: Status 404 returned error can't find the container with id 8c4584cc390448493c2fa5c4e4dbe0203f8c22d4e68a54740c93a7dba966fdaa
Sep 14 03:08:14.874365 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.874326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd" event={"ID":"9ca51ec5-2f3a-4372-92a7-908e2ba739dc","Type":"ContainerStarted","Data":"70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec"}
Sep 14 03:08:14.874725 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.874387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd" event={"ID":"9ca51ec5-2f3a-4372-92a7-908e2ba739dc","Type":"ContainerStarted","Data":"8c4584cc390448493c2fa5c4e4dbe0203f8c22d4e68a54740c93a7dba966fdaa"}
Sep 14 03:08:14.874725 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.874552    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:08:14.893020 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:14.892969    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd" podStartSLOduration=0.892955669 podStartE2EDuration="892.955669ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:08:14.891246762 +0000 UTC m=+961.617204815" watchObservedRunningTime="2026-09-14 03:08:14.892955669 +0000 UTC m=+961.618913720"
Sep 14 03:08:22.461976 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:22.461942    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:22.462472 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:22.462101    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:22.474517 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:22.474494    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:22.931676 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:22.931648    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:08:25.883567 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:25.883537    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:08:27.001285 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.001252    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"]
Sep 14 03:08:27.004877 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.004851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.009377 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.009353    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4eb26b\"/\"kube-root-ca.crt\""
Sep 14 03:08:27.009377 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.009371    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4eb26b\"/\"test-model-4eb26b-kserve-self-signed-certs\""
Sep 14 03:08:27.009810 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.009796    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4eb26b\"/\"default-dockercfg-vzwdf\""
Sep 14 03:08:27.009810 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.009803    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4eb26b\"/\"openshift-service-ca.crt\""
Sep 14 03:08:27.023822 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.023802    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"]
Sep 14 03:08:27.106590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.106566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-home\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.106733 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.106607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-model-cache\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.106733 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.106628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-dshm\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.106856 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.106736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tmp-dir\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.106856 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.106785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tls-certs\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.106856 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.106834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwb8r\" (UniqueName: \"kubernetes.io/projected/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-kube-api-access-jwb8r\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.207443 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.207410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tls-certs\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.207553 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.207454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwb8r\" (UniqueName: \"kubernetes.io/projected/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-kube-api-access-jwb8r\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.207553 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.207506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-home\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.207553 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.207544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-model-cache\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.207685 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.207578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-dshm\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.207685 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.207628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tmp-dir\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.207980 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.207951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-home\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.208094 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.207955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tmp-dir\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.208094 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.207996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-model-cache\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.209803 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.209785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tls-certs\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.209865 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.209799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-dshm\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.224729 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.224705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwb8r\" (UniqueName: \"kubernetes.io/projected/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-kube-api-access-jwb8r\") pod \"test-model-4eb26b-kserve-5b5b79589f-twtrz\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") " pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.313833 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.313809    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:27.589314 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.589290    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"]
Sep 14 03:08:27.590962 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:08:27.590939    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f116f5b_cbb8_4a8f_985c_f32c7f7defbf.slice/crio-c811740466cdc9ef91b9c167b3102c83ecab7d4a26e6daef255220681701a8be WatchSource:0}: Error finding container c811740466cdc9ef91b9c167b3102c83ecab7d4a26e6daef255220681701a8be: Status 404 returned error can't find the container with id c811740466cdc9ef91b9c167b3102c83ecab7d4a26e6daef255220681701a8be
Sep 14 03:08:27.950577 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.950497    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz" event={"ID":"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf","Type":"ContainerStarted","Data":"ce7eb9afdc68ec3524c2e5d11a5b3dfb19fbc0af5d48f4a72bba57fda1119f92"}
Sep 14 03:08:27.950577 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.950537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz" event={"ID":"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf","Type":"ContainerStarted","Data":"c811740466cdc9ef91b9c167b3102c83ecab7d4a26e6daef255220681701a8be"}
Sep 14 03:08:27.973772 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:27.973717    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz" podStartSLOduration=1.973703005 podStartE2EDuration="1.973703005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:08:27.971281918 +0000 UTC m=+974.697239970" watchObservedRunningTime="2026-09-14 03:08:27.973703005 +0000 UTC m=+974.699661057"
Sep 14 03:08:29.058483 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:29.058451    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:08:37.314886 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:37.314811    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:37.314886 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:37.314856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:37.327195 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:37.327170    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:38.021167 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:38.021134    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:08:51.561952 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:08:51.561914    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:09:12.863466 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:12.863435    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:09:29.801261 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.801219    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"]
Sep 14 03:09:29.805041 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.805006    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:09:29.807167 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.807142    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-446c5fbd\""
Sep 14 03:09:29.819584 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.819564    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"]
Sep 14 03:09:29.857896 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.857870    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"]
Sep 14 03:09:29.861059 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.861022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:09:29.863272 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.863253    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-446c5fbd-dockercfg-6zdvf\""
Sep 14 03:09:29.874854 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.874834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"]
Sep 14 03:09:29.976746 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.976704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5720322-1597-4c32-8e0f-12bd316f0156-plugins-config-volume\") pod \"payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m\" (UID: \"a5720322-1597-4c32-8e0f-12bd316f0156\") " pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:09:29.976935 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.976772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696ba33a-15a1-471f-b034-19ada821f616-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425\" (UID: \"696ba33a-15a1-471f-b034-19ada821f616\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:09:29.976935 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.976827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wfz8\" (UniqueName: \"kubernetes.io/projected/a5720322-1597-4c32-8e0f-12bd316f0156-kube-api-access-7wfz8\") pod \"payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m\" (UID: \"a5720322-1597-4c32-8e0f-12bd316f0156\") " pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:09:29.976935 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:29.976923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55f2q\" (UniqueName: \"kubernetes.io/projected/696ba33a-15a1-471f-b034-19ada821f616-kube-api-access-55f2q\") pod \"payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425\" (UID: \"696ba33a-15a1-471f-b034-19ada821f616\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:09:30.077543 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.077473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5720322-1597-4c32-8e0f-12bd316f0156-plugins-config-volume\") pod \"payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m\" (UID: \"a5720322-1597-4c32-8e0f-12bd316f0156\") " pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:09:30.077543 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.077520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696ba33a-15a1-471f-b034-19ada821f616-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425\" (UID: \"696ba33a-15a1-471f-b034-19ada821f616\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:09:30.077763 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.077554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wfz8\" (UniqueName: \"kubernetes.io/projected/a5720322-1597-4c32-8e0f-12bd316f0156-kube-api-access-7wfz8\") pod \"payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m\" (UID: \"a5720322-1597-4c32-8e0f-12bd316f0156\") " pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:09:30.077763 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.077592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55f2q\" (UniqueName: \"kubernetes.io/projected/696ba33a-15a1-471f-b034-19ada821f616-kube-api-access-55f2q\") pod \"payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425\" (UID: \"696ba33a-15a1-471f-b034-19ada821f616\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:09:30.078236 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.078211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5720322-1597-4c32-8e0f-12bd316f0156-plugins-config-volume\") pod \"payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m\" (UID: \"a5720322-1597-4c32-8e0f-12bd316f0156\") " pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:09:30.078348 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.078300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696ba33a-15a1-471f-b034-19ada821f616-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425\" (UID: \"696ba33a-15a1-471f-b034-19ada821f616\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:09:30.085175 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.085157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wfz8\" (UniqueName: \"kubernetes.io/projected/a5720322-1597-4c32-8e0f-12bd316f0156-kube-api-access-7wfz8\") pod \"payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m\" (UID: \"a5720322-1597-4c32-8e0f-12bd316f0156\") " pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:09:30.085257 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.085205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55f2q\" (UniqueName: \"kubernetes.io/projected/696ba33a-15a1-471f-b034-19ada821f616-kube-api-access-55f2q\") pod \"payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425\" (UID: \"696ba33a-15a1-471f-b034-19ada821f616\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:09:30.114615 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.114597    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:09:30.170624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.170397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:09:30.386220 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.386196    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"]
Sep 14 03:09:30.387221 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:09:30.387197    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696ba33a_15a1_471f_b034_19ada821f616.slice/crio-5cfca9c6d9846eb6b865fd482682bb89c33aabe5d50c7739b4f32d984f25e045 WatchSource:0}: Error finding container 5cfca9c6d9846eb6b865fd482682bb89c33aabe5d50c7739b4f32d984f25e045: Status 404 returned error can't find the container with id 5cfca9c6d9846eb6b865fd482682bb89c33aabe5d50c7739b4f32d984f25e045
Sep 14 03:09:30.448059 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:30.445921    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"]
Sep 14 03:09:31.328822 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:31.328785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m" event={"ID":"a5720322-1597-4c32-8e0f-12bd316f0156","Type":"ContainerStarted","Data":"ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a"}
Sep 14 03:09:31.329339 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:31.328946    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:09:31.329339 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:31.328972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m" event={"ID":"a5720322-1597-4c32-8e0f-12bd316f0156","Type":"ContainerStarted","Data":"3a630e1deb92f709d4c33093ded71af5355afc7809fb2c5202c1a6088881858d"}
Sep 14 03:09:31.330534 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:31.330509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425" event={"ID":"696ba33a-15a1-471f-b034-19ada821f616","Type":"ContainerStarted","Data":"bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e"}
Sep 14 03:09:31.330651 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:31.330545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425" event={"ID":"696ba33a-15a1-471f-b034-19ada821f616","Type":"ContainerStarted","Data":"5cfca9c6d9846eb6b865fd482682bb89c33aabe5d50c7739b4f32d984f25e045"}
Sep 14 03:09:31.330651 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:31.330586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:09:31.349005 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:31.348955    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m" podStartSLOduration=2.348938665 podStartE2EDuration="2.348938665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:09:31.346887429 +0000 UTC m=+1038.072845485" watchObservedRunningTime="2026-09-14 03:09:31.348938665 +0000 UTC m=+1038.074896718"
Sep 14 03:09:31.363607 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:31.363567    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425" podStartSLOduration=2.363553152 podStartE2EDuration="2.363553152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:09:31.362395207 +0000 UTC m=+1038.088353262" watchObservedRunningTime="2026-09-14 03:09:31.363553152 +0000 UTC m=+1038.089511204"
Sep 14 03:09:38.566999 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:38.566966    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:09:42.338167 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:42.338138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:09:42.338532 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:42.338205    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:09:50.065226 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:50.065193    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"]
Sep 14 03:09:50.068761 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:50.068744    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"
Sep 14 03:09:50.070701 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:50.070677    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:09:50.070823 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:50.070765    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sv5sd\""
Sep 14 03:09:50.075558 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:50.075534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"]
Sep 14 03:09:50.148929 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:50.148903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d20035-f7f5-47e8-b022-2b709f23be37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh\" (UID: \"40d20035-f7f5-47e8-b022-2b709f23be37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"
Sep 14 03:09:50.250271 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:50.250241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d20035-f7f5-47e8-b022-2b709f23be37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh\" (UID: \"40d20035-f7f5-47e8-b022-2b709f23be37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"
Sep 14 03:09:50.250879 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:50.250859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d20035-f7f5-47e8-b022-2b709f23be37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh\" (UID: \"40d20035-f7f5-47e8-b022-2b709f23be37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"
Sep 14 03:09:50.381539 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:50.381454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"
Sep 14 03:09:50.655692 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:50.655661    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"]
Sep 14 03:09:50.657131 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:09:50.657094    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d20035_f7f5_47e8_b022_2b709f23be37.slice/crio-5129f989af85b1afe93ad212c30593264cefa9fb871674256c9295a80bbd1238 WatchSource:0}: Error finding container 5129f989af85b1afe93ad212c30593264cefa9fb871674256c9295a80bbd1238: Status 404 returned error can't find the container with id 5129f989af85b1afe93ad212c30593264cefa9fb871674256c9295a80bbd1238
Sep 14 03:09:51.449755 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:51.449725    2580 generic.go:356] "Generic (PLEG): container finished" podID="40d20035-f7f5-47e8-b022-2b709f23be37" containerID="51d9f0a4d2890cb0049bb220db4316859db1abe1c96884e535b6fc1b82c9ced2" exitCode=0
Sep 14 03:09:51.450115 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:51.449771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh" event={"ID":"40d20035-f7f5-47e8-b022-2b709f23be37","Type":"ContainerDied","Data":"51d9f0a4d2890cb0049bb220db4316859db1abe1c96884e535b6fc1b82c9ced2"}
Sep 14 03:09:51.450115 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:51.449823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh" event={"ID":"40d20035-f7f5-47e8-b022-2b709f23be37","Type":"ContainerStarted","Data":"5129f989af85b1afe93ad212c30593264cefa9fb871674256c9295a80bbd1238"}
Sep 14 03:09:52.587673 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:52.587648    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"
Sep 14 03:09:52.676719 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:52.676691    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d20035-f7f5-47e8-b022-2b709f23be37-maas-api-service-ca\") pod \"40d20035-f7f5-47e8-b022-2b709f23be37\" (UID: \"40d20035-f7f5-47e8-b022-2b709f23be37\") "
Sep 14 03:09:52.677060 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:52.677024    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d20035-f7f5-47e8-b022-2b709f23be37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40d20035-f7f5-47e8-b022-2b709f23be37" (UID: "40d20035-f7f5-47e8-b022-2b709f23be37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:09:52.777904 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:52.777881    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d20035-f7f5-47e8-b022-2b709f23be37-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:09:53.463815 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:53.463784    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"
Sep 14 03:09:53.463997 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:53.463810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh" event={"ID":"40d20035-f7f5-47e8-b022-2b709f23be37","Type":"ContainerDied","Data":"5129f989af85b1afe93ad212c30593264cefa9fb871674256c9295a80bbd1238"}
Sep 14 03:09:53.463997 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:09:53.463859    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5129f989af85b1afe93ad212c30593264cefa9fb871674256c9295a80bbd1238"
Sep 14 03:10:01.020308 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.020279    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"]
Sep 14 03:10:01.020826 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.020516    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd" podUID="9ca51ec5-2f3a-4372-92a7-908e2ba739dc" containerName="payload-processing" containerID="cri-o://70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec" gracePeriod=30
Sep 14 03:10:01.202139 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.202108    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"]
Sep 14 03:10:01.204587 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.204560    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8364a0-fbfxh"]
Sep 14 03:10:01.275240 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.275185    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:10:01.351404 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.350537    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqkp6\" (UniqueName: \"kubernetes.io/projected/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-kube-api-access-gqkp6\") pod \"9ca51ec5-2f3a-4372-92a7-908e2ba739dc\" (UID: \"9ca51ec5-2f3a-4372-92a7-908e2ba739dc\") "
Sep 14 03:10:01.351404 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.350707    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-plugins-config-volume\") pod \"9ca51ec5-2f3a-4372-92a7-908e2ba739dc\" (UID: \"9ca51ec5-2f3a-4372-92a7-908e2ba739dc\") "
Sep 14 03:10:01.351404 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.351390    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ca51ec5-2f3a-4372-92a7-908e2ba739dc" (UID: "9ca51ec5-2f3a-4372-92a7-908e2ba739dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:10:01.356701 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.351527    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:10:01.356701 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.353766    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-kube-api-access-gqkp6" (OuterVolumeSpecName: "kube-api-access-gqkp6") pod "9ca51ec5-2f3a-4372-92a7-908e2ba739dc" (UID: "9ca51ec5-2f3a-4372-92a7-908e2ba739dc"). InnerVolumeSpecName "kube-api-access-gqkp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:10:01.452822 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.452792    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqkp6\" (UniqueName: \"kubernetes.io/projected/9ca51ec5-2f3a-4372-92a7-908e2ba739dc-kube-api-access-gqkp6\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:10:01.516157 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.516125    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ca51ec5-2f3a-4372-92a7-908e2ba739dc" containerID="70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec" exitCode=0
Sep 14 03:10:01.516289 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.516169    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"
Sep 14 03:10:01.516289 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.516212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd" event={"ID":"9ca51ec5-2f3a-4372-92a7-908e2ba739dc","Type":"ContainerDied","Data":"70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec"}
Sep 14 03:10:01.516289 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.516263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd" event={"ID":"9ca51ec5-2f3a-4372-92a7-908e2ba739dc","Type":"ContainerDied","Data":"8c4584cc390448493c2fa5c4e4dbe0203f8c22d4e68a54740c93a7dba966fdaa"}
Sep 14 03:10:01.516289 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.516285    2580 scope.go:117] "RemoveContainer" containerID="70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec"
Sep 14 03:10:01.526041 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.526017    2580 scope.go:117] "RemoveContainer" containerID="70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec"
Sep 14 03:10:01.526307 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:10:01.526290    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec\": container with ID starting with 70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec not found: ID does not exist" containerID="70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec"
Sep 14 03:10:01.526355 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.526316    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec"} err="failed to get container status \"70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec\": rpc error: code = NotFound desc = could not find container \"70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec\": container with ID starting with 70445f52bc902c9e037178923d20a945a9c7a1da5ae0208e175d2cee4321e9ec not found: ID does not exist"
Sep 14 03:10:01.538231 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.538207    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"]
Sep 14 03:10:01.547518 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.547497    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8364a0-7f467ccc8f-cr8cd"]
Sep 14 03:10:01.757199 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.757167    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d20035-f7f5-47e8-b022-2b709f23be37" path="/var/lib/kubelet/pods/40d20035-f7f5-47e8-b022-2b709f23be37/volumes"
Sep 14 03:10:01.757514 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:01.757501    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ca51ec5-2f3a-4372-92a7-908e2ba739dc" path="/var/lib/kubelet/pods/9ca51ec5-2f3a-4372-92a7-908e2ba739dc/volumes"
Sep 14 03:10:04.696442 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.696405    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"]
Sep 14 03:10:04.696803 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.696786    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca51ec5-2f3a-4372-92a7-908e2ba739dc" containerName="payload-processing"
Sep 14 03:10:04.696803 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.696797    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca51ec5-2f3a-4372-92a7-908e2ba739dc" containerName="payload-processing"
Sep 14 03:10:04.696876 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.696813    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d20035-f7f5-47e8-b022-2b709f23be37" containerName="revoke-keys"
Sep 14 03:10:04.696876 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.696819    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d20035-f7f5-47e8-b022-2b709f23be37" containerName="revoke-keys"
Sep 14 03:10:04.696948 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.696885    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d20035-f7f5-47e8-b022-2b709f23be37" containerName="revoke-keys"
Sep 14 03:10:04.696948 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.696898    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ca51ec5-2f3a-4372-92a7-908e2ba739dc" containerName="payload-processing"
Sep 14 03:10:04.701170 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.701152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"
Sep 14 03:10:04.704570 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.704549    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:10:04.704570 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.704565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sv5sd\""
Sep 14 03:10:04.712306 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.712282    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"]
Sep 14 03:10:04.788935 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.788902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1945ab-8a0a-4615-8366-079c711d744c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp\" (UID: \"3b1945ab-8a0a-4615-8366-079c711d744c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"
Sep 14 03:10:04.889576 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.889540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1945ab-8a0a-4615-8366-079c711d744c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp\" (UID: \"3b1945ab-8a0a-4615-8366-079c711d744c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"
Sep 14 03:10:04.890228 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:04.890208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1945ab-8a0a-4615-8366-079c711d744c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp\" (UID: \"3b1945ab-8a0a-4615-8366-079c711d744c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"
Sep 14 03:10:05.011577 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:05.011547    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"
Sep 14 03:10:05.286223 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:05.286200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"]
Sep 14 03:10:05.287380 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:10:05.287350    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1945ab_8a0a_4615_8366_079c711d744c.slice/crio-41da8c5247a707ef4affdd3d53da0ffcb08b90ef971f53b30d8fc26d44a8691e WatchSource:0}: Error finding container 41da8c5247a707ef4affdd3d53da0ffcb08b90ef971f53b30d8fc26d44a8691e: Status 404 returned error can't find the container with id 41da8c5247a707ef4affdd3d53da0ffcb08b90ef971f53b30d8fc26d44a8691e
Sep 14 03:10:05.545518 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:05.545448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp" event={"ID":"3b1945ab-8a0a-4615-8366-079c711d744c","Type":"ContainerStarted","Data":"41da8c5247a707ef4affdd3d53da0ffcb08b90ef971f53b30d8fc26d44a8691e"}
Sep 14 03:10:06.554150 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:06.554116    2580 generic.go:356] "Generic (PLEG): container finished" podID="3b1945ab-8a0a-4615-8366-079c711d744c" containerID="3c6fe9ad9aa3393d283cd59fb13fd14fa9b3bbfd1a6b2e22b73d5f010653a207" exitCode=0
Sep 14 03:10:06.554511 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:06.554222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp" event={"ID":"3b1945ab-8a0a-4615-8366-079c711d744c","Type":"ContainerDied","Data":"3c6fe9ad9aa3393d283cd59fb13fd14fa9b3bbfd1a6b2e22b73d5f010653a207"}
Sep 14 03:10:07.708170 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:07.708144    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"
Sep 14 03:10:07.820210 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:07.820131    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1945ab-8a0a-4615-8366-079c711d744c-maas-api-service-ca\") pod \"3b1945ab-8a0a-4615-8366-079c711d744c\" (UID: \"3b1945ab-8a0a-4615-8366-079c711d744c\") "
Sep 14 03:10:07.820453 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:07.820431    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1945ab-8a0a-4615-8366-079c711d744c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b1945ab-8a0a-4615-8366-079c711d744c" (UID: "3b1945ab-8a0a-4615-8366-079c711d744c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:10:07.820535 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:07.820523    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1945ab-8a0a-4615-8366-079c711d744c-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:10:08.571235 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:08.571206    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"
Sep 14 03:10:08.571496 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:08.571288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp" event={"ID":"3b1945ab-8a0a-4615-8366-079c711d744c","Type":"ContainerDied","Data":"41da8c5247a707ef4affdd3d53da0ffcb08b90ef971f53b30d8fc26d44a8691e"}
Sep 14 03:10:08.571645 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:08.571523    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41da8c5247a707ef4affdd3d53da0ffcb08b90ef971f53b30d8fc26d44a8691e"
Sep 14 03:10:15.517813 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.517780    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"]
Sep 14 03:10:15.518509 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.518004    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m" podUID="a5720322-1597-4c32-8e0f-12bd316f0156" containerName="payload-processing" containerID="cri-o://ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a" gracePeriod=30
Sep 14 03:10:15.528378 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.528352    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"]
Sep 14 03:10:15.528916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.528609    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425" podUID="696ba33a-15a1-471f-b034-19ada821f616" containerName="payload-pre-processing" containerID="cri-o://bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e" gracePeriod=30
Sep 14 03:10:15.672898 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.672852    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"]
Sep 14 03:10:15.677128 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.677101    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-446c5fbd-89fxp"]
Sep 14 03:10:15.759979 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.759946    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1945ab-8a0a-4615-8366-079c711d744c" path="/var/lib/kubelet/pods/3b1945ab-8a0a-4615-8366-079c711d744c/volumes"
Sep 14 03:10:15.802718 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.802696    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:10:15.805869 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.805850    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:10:15.893499 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.893465    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55f2q\" (UniqueName: \"kubernetes.io/projected/696ba33a-15a1-471f-b034-19ada821f616-kube-api-access-55f2q\") pod \"696ba33a-15a1-471f-b034-19ada821f616\" (UID: \"696ba33a-15a1-471f-b034-19ada821f616\") "
Sep 14 03:10:15.893499 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.893502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5720322-1597-4c32-8e0f-12bd316f0156-plugins-config-volume\") pod \"a5720322-1597-4c32-8e0f-12bd316f0156\" (UID: \"a5720322-1597-4c32-8e0f-12bd316f0156\") "
Sep 14 03:10:15.893754 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.893521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wfz8\" (UniqueName: \"kubernetes.io/projected/a5720322-1597-4c32-8e0f-12bd316f0156-kube-api-access-7wfz8\") pod \"a5720322-1597-4c32-8e0f-12bd316f0156\" (UID: \"a5720322-1597-4c32-8e0f-12bd316f0156\") "
Sep 14 03:10:15.893754 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.893599    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696ba33a-15a1-471f-b034-19ada821f616-plugins-config-volume\") pod \"696ba33a-15a1-471f-b034-19ada821f616\" (UID: \"696ba33a-15a1-471f-b034-19ada821f616\") "
Sep 14 03:10:15.894098 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.894066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5720322-1597-4c32-8e0f-12bd316f0156-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5720322-1597-4c32-8e0f-12bd316f0156" (UID: "a5720322-1597-4c32-8e0f-12bd316f0156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:10:15.894243 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.894130    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696ba33a-15a1-471f-b034-19ada821f616-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "696ba33a-15a1-471f-b034-19ada821f616" (UID: "696ba33a-15a1-471f-b034-19ada821f616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:10:15.896126 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.896093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696ba33a-15a1-471f-b034-19ada821f616-kube-api-access-55f2q" (OuterVolumeSpecName: "kube-api-access-55f2q") pod "696ba33a-15a1-471f-b034-19ada821f616" (UID: "696ba33a-15a1-471f-b034-19ada821f616"). InnerVolumeSpecName "kube-api-access-55f2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:10:15.896224 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.896143    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5720322-1597-4c32-8e0f-12bd316f0156-kube-api-access-7wfz8" (OuterVolumeSpecName: "kube-api-access-7wfz8") pod "a5720322-1597-4c32-8e0f-12bd316f0156" (UID: "a5720322-1597-4c32-8e0f-12bd316f0156"). InnerVolumeSpecName "kube-api-access-7wfz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:10:15.995483 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.995440    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55f2q\" (UniqueName: \"kubernetes.io/projected/696ba33a-15a1-471f-b034-19ada821f616-kube-api-access-55f2q\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:10:15.995483 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.995468    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5720322-1597-4c32-8e0f-12bd316f0156-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:10:15.995483 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.995478    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wfz8\" (UniqueName: \"kubernetes.io/projected/a5720322-1597-4c32-8e0f-12bd316f0156-kube-api-access-7wfz8\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:10:15.995483 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:15.995489    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696ba33a-15a1-471f-b034-19ada821f616-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:10:16.623138 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.623107    2580 generic.go:356] "Generic (PLEG): container finished" podID="696ba33a-15a1-471f-b034-19ada821f616" containerID="bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e" exitCode=0
Sep 14 03:10:16.623531 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.623141    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"
Sep 14 03:10:16.623531 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.623235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425" event={"ID":"696ba33a-15a1-471f-b034-19ada821f616","Type":"ContainerDied","Data":"bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e"}
Sep 14 03:10:16.623531 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.623287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425" event={"ID":"696ba33a-15a1-471f-b034-19ada821f616","Type":"ContainerDied","Data":"5cfca9c6d9846eb6b865fd482682bb89c33aabe5d50c7739b4f32d984f25e045"}
Sep 14 03:10:16.623531 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.623313    2580 scope.go:117] "RemoveContainer" containerID="bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e"
Sep 14 03:10:16.624824 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.624801    2580 generic.go:356] "Generic (PLEG): container finished" podID="a5720322-1597-4c32-8e0f-12bd316f0156" containerID="ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a" exitCode=0
Sep 14 03:10:16.624900 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.624857    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"
Sep 14 03:10:16.624900 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.624887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m" event={"ID":"a5720322-1597-4c32-8e0f-12bd316f0156","Type":"ContainerDied","Data":"ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a"}
Sep 14 03:10:16.625000 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.624935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m" event={"ID":"a5720322-1597-4c32-8e0f-12bd316f0156","Type":"ContainerDied","Data":"3a630e1deb92f709d4c33093ded71af5355afc7809fb2c5202c1a6088881858d"}
Sep 14 03:10:16.636858 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.636842    2580 scope.go:117] "RemoveContainer" containerID="bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e"
Sep 14 03:10:16.637163 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:10:16.637138    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e\": container with ID starting with bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e not found: ID does not exist" containerID="bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e"
Sep 14 03:10:16.637261 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.637171    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e"} err="failed to get container status \"bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e\": rpc error: code = NotFound desc = could not find container \"bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e\": container with ID starting with bd07d02cca1a7ffb5633cf0b46783c4c3075d36bcb5ee6aa1eba790e30c4e05e not found: ID does not exist"
Sep 14 03:10:16.637261 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.637193    2580 scope.go:117] "RemoveContainer" containerID="ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a"
Sep 14 03:10:16.645452 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.645430    2580 scope.go:117] "RemoveContainer" containerID="ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a"
Sep 14 03:10:16.646214 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:10:16.646162    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a\": container with ID starting with ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a not found: ID does not exist" containerID="ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a"
Sep 14 03:10:16.646317 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.646221    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a"} err="failed to get container status \"ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a\": rpc error: code = NotFound desc = could not find container \"ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a\": container with ID starting with ef572fd4dbfe33147d40396a66e2f19dd0aba47acc895c4f1826fc8ad2109c2a not found: ID does not exist"
Sep 14 03:10:16.649050 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.649017    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"]
Sep 14 03:10:16.650363 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:10:16.650340    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5720322_1597_4c32_8e0f_12bd316f0156.slice/crio-3a630e1deb92f709d4c33093ded71af5355afc7809fb2c5202c1a6088881858d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696ba33a_15a1_471f_b034_19ada821f616.slice\": RecentStats: unable to find data in memory cache]"
Sep 14 03:10:16.654579 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.654558    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-446c5fbd-684c8685bb-qm425"]
Sep 14 03:10:16.664400 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.664379    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"]
Sep 14 03:10:16.673688 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:16.673668    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-446c5fbd-775749b5c7-sv46m"]
Sep 14 03:10:17.119020 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:17.118985    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:10:17.756530 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:17.756496    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="696ba33a-15a1-471f-b034-19ada821f616" path="/var/lib/kubelet/pods/696ba33a-15a1-471f-b034-19ada821f616/volumes"
Sep 14 03:10:17.756889 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:17.756864    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5720322-1597-4c32-8e0f-12bd316f0156" path="/var/lib/kubelet/pods/a5720322-1597-4c32-8e0f-12bd316f0156/volumes"
Sep 14 03:10:26.643200 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.643165    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"]
Sep 14 03:10:26.643606 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.643513    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696ba33a-15a1-471f-b034-19ada821f616" containerName="payload-pre-processing"
Sep 14 03:10:26.643606 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.643525    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="696ba33a-15a1-471f-b034-19ada821f616" containerName="payload-pre-processing"
Sep 14 03:10:26.643606 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.643539    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1945ab-8a0a-4615-8366-079c711d744c" containerName="revoke-keys"
Sep 14 03:10:26.643606 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.643545    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1945ab-8a0a-4615-8366-079c711d744c" containerName="revoke-keys"
Sep 14 03:10:26.643606 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.643558    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5720322-1597-4c32-8e0f-12bd316f0156" containerName="payload-processing"
Sep 14 03:10:26.643606 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.643563    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5720322-1597-4c32-8e0f-12bd316f0156" containerName="payload-processing"
Sep 14 03:10:26.643827 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.643640    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1945ab-8a0a-4615-8366-079c711d744c" containerName="revoke-keys"
Sep 14 03:10:26.643827 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.643650    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="696ba33a-15a1-471f-b034-19ada821f616" containerName="payload-pre-processing"
Sep 14 03:10:26.643827 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.643659    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5720322-1597-4c32-8e0f-12bd316f0156" containerName="payload-processing"
Sep 14 03:10:26.647967 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.647951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.649925 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.649904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a752f\"/\"distinct-w0-3a752f-kserve-self-signed-certs\""
Sep 14 03:10:26.658695 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.658672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"]
Sep 14 03:10:26.693935 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.693905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-dshm\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.694104 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.693954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-home\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.694104 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.693983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tmp-dir\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.694104 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.694058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-model-cache\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.694104 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.694079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tls-certs\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.694292 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.694113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqsmv\" (UniqueName: \"kubernetes.io/projected/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-kube-api-access-qqsmv\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.795482 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.795451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-home\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.795612 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.795508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tmp-dir\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.795612 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.795581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-model-cache\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.795723 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.795615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tls-certs\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.795723 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.795675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqsmv\" (UniqueName: \"kubernetes.io/projected/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-kube-api-access-qqsmv\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.795835 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.795773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-dshm\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.795896 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.795877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-home\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.795953 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.795893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tmp-dir\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.796246 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.796225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-model-cache\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.798010 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.797991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tls-certs\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.798127 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.798099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-dshm\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.804328 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.804306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqsmv\" (UniqueName: \"kubernetes.io/projected/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-kube-api-access-qqsmv\") pod \"distinct-w0-3a752f-kserve-7896cc46c9-kk7bg\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:26.958367 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:26.958281    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:27.231691 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:27.231664    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"]
Sep 14 03:10:27.233427 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:10:27.233399    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6ee7c3f_7b1e_492e_b507_f5ed6e68d5cf.slice/crio-377d500f1b2fe8d4924471e56ae474b415207a49d8bf649dffc371add913762c WatchSource:0}: Error finding container 377d500f1b2fe8d4924471e56ae474b415207a49d8bf649dffc371add913762c: Status 404 returned error can't find the container with id 377d500f1b2fe8d4924471e56ae474b415207a49d8bf649dffc371add913762c
Sep 14 03:10:27.700682 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:27.700639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg" event={"ID":"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf","Type":"ContainerStarted","Data":"0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65"}
Sep 14 03:10:27.701079 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:27.700690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg" event={"ID":"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf","Type":"ContainerStarted","Data":"377d500f1b2fe8d4924471e56ae474b415207a49d8bf649dffc371add913762c"}
Sep 14 03:10:27.745749 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:27.745695    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg" podStartSLOduration=1.745680007 podStartE2EDuration="1.745680007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:10:27.741155815 +0000 UTC m=+1094.467113887" watchObservedRunningTime="2026-09-14 03:10:27.745680007 +0000 UTC m=+1094.471638080"
Sep 14 03:10:36.958728 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:36.958694    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:36.959274 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:36.958741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:36.971407 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:36.971384    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:37.778219 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:37.778188    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:10:43.180231 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.180190    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"]
Sep 14 03:10:43.183888 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.183866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.186445 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.186420    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 03:10:43.186561 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.186545    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:10:43.187258 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.187237    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 03:10:43.187414 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.187254    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-ce5b9b-serving-cert\""
Sep 14 03:10:43.187414 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.187327    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jf78h\""
Sep 14 03:10:43.199900 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.199880    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"]
Sep 14 03:10:43.356442 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.356414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.356590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.356449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph74v\" (UniqueName: \"kubernetes.io/projected/9c07fba8-b58c-424d-8861-7da5cce15c8e-kube-api-access-ph74v\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.356590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.356543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.356665 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.356601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c07fba8-b58c-424d-8861-7da5cce15c8e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.457529 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.457446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.457529 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.457497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c07fba8-b58c-424d-8861-7da5cce15c8e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.457711 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.457546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.457711 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.457567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph74v\" (UniqueName: \"kubernetes.io/projected/9c07fba8-b58c-424d-8861-7da5cce15c8e-kube-api-access-ph74v\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.458288 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.458262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c07fba8-b58c-424d-8861-7da5cce15c8e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.459874 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.459841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.459983 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.459898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.466374 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.466357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph74v\" (UniqueName: \"kubernetes.io/projected/9c07fba8-b58c-424d-8861-7da5cce15c8e-kube-api-access-ph74v\") pod \"maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.496554 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.496529    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:43.771232 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.771202    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"]
Sep 14 03:10:43.771905 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:10:43.771879    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c07fba8_b58c_424d_8861_7da5cce15c8e.slice/crio-df1af8dc90f783b3f21c32d11f8844baae19585f46f192894d604105818cbd36 WatchSource:0}: Error finding container df1af8dc90f783b3f21c32d11f8844baae19585f46f192894d604105818cbd36: Status 404 returned error can't find the container with id df1af8dc90f783b3f21c32d11f8844baae19585f46f192894d604105818cbd36
Sep 14 03:10:43.773267 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.773248    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 03:10:43.802512 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:43.802484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82" event={"ID":"9c07fba8-b58c-424d-8861-7da5cce15c8e","Type":"ContainerStarted","Data":"df1af8dc90f783b3f21c32d11f8844baae19585f46f192894d604105818cbd36"}
Sep 14 03:10:44.810093 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:44.810054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82" event={"ID":"9c07fba8-b58c-424d-8861-7da5cce15c8e","Type":"ContainerStarted","Data":"d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64"}
Sep 14 03:10:44.810457 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:44.810116    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:44.827637 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:44.827596    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82" podStartSLOduration=1.45029065 podStartE2EDuration="1.827578312s" totalImagesPullingTime="377.287662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:10:44.826381543 +0000 UTC m=+1111.552339597" watchObservedRunningTime="2026-09-14 03:10:44.827578312 +0000 UTC m=+1111.553536364"
Sep 14 03:10:50.821340 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:50.821311    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:10:50.960859 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:10:50.960824    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:11:18.132601 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.132568    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"]
Sep 14 03:11:18.137244 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.137225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.140670 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.140646    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-17d8b8-serving-cert\""
Sep 14 03:11:18.152270 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.152248    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"]
Sep 14 03:11:18.282637 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.282608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.282823 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.282645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17441f5d-c1ae-41ed-acc1-1e3075d91234-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.282823 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.282671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6dvp\" (UniqueName: \"kubernetes.io/projected/17441f5d-c1ae-41ed-acc1-1e3075d91234-kube-api-access-z6dvp\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.282823 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.282756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.384091 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.383993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17441f5d-c1ae-41ed-acc1-1e3075d91234-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.384091 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.384062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6dvp\" (UniqueName: \"kubernetes.io/projected/17441f5d-c1ae-41ed-acc1-1e3075d91234-kube-api-access-z6dvp\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.384269 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.384114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.384269 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.384248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.384768 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.384741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17441f5d-c1ae-41ed-acc1-1e3075d91234-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.386539 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.386512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.386666 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.386602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.396142 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.396116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6dvp\" (UniqueName: \"kubernetes.io/projected/17441f5d-c1ae-41ed-acc1-1e3075d91234-kube-api-access-z6dvp\") pod \"maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.447143 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.447104    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:18.734289 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.734214    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"]
Sep 14 03:11:18.737167 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:11:18.737140    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17441f5d_c1ae_41ed_acc1_1e3075d91234.slice/crio-dc9115ce0d76a0923fcc9b275d759266912910313c34c9de9c92e9c200fca4da WatchSource:0}: Error finding container dc9115ce0d76a0923fcc9b275d759266912910313c34c9de9c92e9c200fca4da: Status 404 returned error can't find the container with id dc9115ce0d76a0923fcc9b275d759266912910313c34c9de9c92e9c200fca4da
Sep 14 03:11:18.993093 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:18.992986    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"]
Sep 14 03:11:19.004194 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.004166    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:11:19.007602 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.006839    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-17d8b8-dockercfg-z64v9\""
Sep 14 03:11:19.007602 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.007387    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-17d8b8\""
Sep 14 03:11:19.008941 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.008916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"]
Sep 14 03:11:19.014867 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.014820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96" event={"ID":"17441f5d-c1ae-41ed-acc1-1e3075d91234","Type":"ContainerStarted","Data":"dc9115ce0d76a0923fcc9b275d759266912910313c34c9de9c92e9c200fca4da"}
Sep 14 03:11:19.092231 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.092206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkd56\" (UniqueName: \"kubernetes.io/projected/65bf6740-dca5-4443-8ac1-79500cb5427c-kube-api-access-wkd56\") pod \"payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x\" (UID: \"65bf6740-dca5-4443-8ac1-79500cb5427c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:11:19.092355 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.092310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bf6740-dca5-4443-8ac1-79500cb5427c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x\" (UID: \"65bf6740-dca5-4443-8ac1-79500cb5427c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:11:19.193449 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.193415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bf6740-dca5-4443-8ac1-79500cb5427c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x\" (UID: \"65bf6740-dca5-4443-8ac1-79500cb5427c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:11:19.193857 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.193556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd56\" (UniqueName: \"kubernetes.io/projected/65bf6740-dca5-4443-8ac1-79500cb5427c-kube-api-access-wkd56\") pod \"payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x\" (UID: \"65bf6740-dca5-4443-8ac1-79500cb5427c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:11:19.194198 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.194175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bf6740-dca5-4443-8ac1-79500cb5427c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x\" (UID: \"65bf6740-dca5-4443-8ac1-79500cb5427c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:11:19.202493 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.202471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkd56\" (UniqueName: \"kubernetes.io/projected/65bf6740-dca5-4443-8ac1-79500cb5427c-kube-api-access-wkd56\") pod \"payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x\" (UID: \"65bf6740-dca5-4443-8ac1-79500cb5427c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:11:19.319546 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.319518    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:11:19.595702 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:19.595669    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"]
Sep 14 03:11:19.597335 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:11:19.597305    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bf6740_dca5_4443_8ac1_79500cb5427c.slice/crio-094bc49087ee068b39989ad00a20bf61963cb2f6973276924b0d1586946f6a06 WatchSource:0}: Error finding container 094bc49087ee068b39989ad00a20bf61963cb2f6973276924b0d1586946f6a06: Status 404 returned error can't find the container with id 094bc49087ee068b39989ad00a20bf61963cb2f6973276924b0d1586946f6a06
Sep 14 03:11:20.022603 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:20.022564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96" event={"ID":"17441f5d-c1ae-41ed-acc1-1e3075d91234","Type":"ContainerStarted","Data":"fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6"}
Sep 14 03:11:20.022783 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:20.022630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:20.024295 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:20.024269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x" event={"ID":"65bf6740-dca5-4443-8ac1-79500cb5427c","Type":"ContainerStarted","Data":"007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0"}
Sep 14 03:11:20.024402 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:20.024314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x" event={"ID":"65bf6740-dca5-4443-8ac1-79500cb5427c","Type":"ContainerStarted","Data":"094bc49087ee068b39989ad00a20bf61963cb2f6973276924b0d1586946f6a06"}
Sep 14 03:11:20.024402 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:20.024359    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:11:20.042135 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:20.042096    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96" podStartSLOduration=1.685032366 podStartE2EDuration="2.04208174s" totalImagesPullingTime="357.049374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:20.039402007 +0000 UTC m=+1146.765360063" watchObservedRunningTime="2026-09-14 03:11:20.04208174 +0000 UTC m=+1146.768039791"
Sep 14 03:11:20.055568 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:20.055523    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x" podStartSLOduration=2.055507255 podStartE2EDuration="2.055507255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:20.054460947 +0000 UTC m=+1146.780418999" watchObservedRunningTime="2026-09-14 03:11:20.055507255 +0000 UTC m=+1146.781465308"
Sep 14 03:11:21.952933 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:21.952900    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"]
Sep 14 03:11:23.123085 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:23.123055    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"]
Sep 14 03:11:23.123569 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:23.123251    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx" podUID="763ec406-383b-458d-af55-2b45a337efa5" containerName="payload-pre-processing" containerID="cri-o://68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be" gracePeriod=30
Sep 14 03:11:23.372002 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:23.371977    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:11:23.536022 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:23.535995    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4xz4\" (UniqueName: \"kubernetes.io/projected/763ec406-383b-458d-af55-2b45a337efa5-kube-api-access-p4xz4\") pod \"763ec406-383b-458d-af55-2b45a337efa5\" (UID: \"763ec406-383b-458d-af55-2b45a337efa5\") "
Sep 14 03:11:23.536202 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:23.536070    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ec406-383b-458d-af55-2b45a337efa5-plugins-config-volume\") pod \"763ec406-383b-458d-af55-2b45a337efa5\" (UID: \"763ec406-383b-458d-af55-2b45a337efa5\") "
Sep 14 03:11:23.536428 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:23.536405    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763ec406-383b-458d-af55-2b45a337efa5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "763ec406-383b-458d-af55-2b45a337efa5" (UID: "763ec406-383b-458d-af55-2b45a337efa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:11:23.538123 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:23.538097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763ec406-383b-458d-af55-2b45a337efa5-kube-api-access-p4xz4" (OuterVolumeSpecName: "kube-api-access-p4xz4") pod "763ec406-383b-458d-af55-2b45a337efa5" (UID: "763ec406-383b-458d-af55-2b45a337efa5"). InnerVolumeSpecName "kube-api-access-p4xz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:11:23.637474 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:23.637441    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4xz4\" (UniqueName: \"kubernetes.io/projected/763ec406-383b-458d-af55-2b45a337efa5-kube-api-access-p4xz4\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:11:23.637474 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:23.637471    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ec406-383b-458d-af55-2b45a337efa5-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:11:24.052021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:24.051985    2580 generic.go:356] "Generic (PLEG): container finished" podID="763ec406-383b-458d-af55-2b45a337efa5" containerID="68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be" exitCode=0
Sep 14 03:11:24.052181 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:24.052045    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"
Sep 14 03:11:24.052181 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:24.052079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx" event={"ID":"763ec406-383b-458d-af55-2b45a337efa5","Type":"ContainerDied","Data":"68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be"}
Sep 14 03:11:24.052181 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:24.052135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx" event={"ID":"763ec406-383b-458d-af55-2b45a337efa5","Type":"ContainerDied","Data":"793aaa062828d06c5a00cf4e5263022f8c8ae722fa26367c27c3a059669a8609"}
Sep 14 03:11:24.052181 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:24.052161    2580 scope.go:117] "RemoveContainer" containerID="68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be"
Sep 14 03:11:24.061282 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:24.061265    2580 scope.go:117] "RemoveContainer" containerID="68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be"
Sep 14 03:11:24.061541 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:11:24.061523    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be\": container with ID starting with 68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be not found: ID does not exist" containerID="68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be"
Sep 14 03:11:24.061591 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:24.061548    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be"} err="failed to get container status \"68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be\": rpc error: code = NotFound desc = could not find container \"68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be\": container with ID starting with 68a222268115632c382bba5755fa011e91114014c72b63ed0315066cbd5703be not found: ID does not exist"
Sep 14 03:11:24.069891 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:24.069866    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"]
Sep 14 03:11:24.078470 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:24.078452    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f002a1-7b4b649b54-wcxmx"]
Sep 14 03:11:25.757638 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:25.757605    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="763ec406-383b-458d-af55-2b45a337efa5" path="/var/lib/kubelet/pods/763ec406-383b-458d-af55-2b45a337efa5/volumes"
Sep 14 03:11:26.035872 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:26.035792    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:11:27.259411 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:27.259381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:11:31.032499 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:31.032471    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:11:36.971687 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:36.971589    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp" podUID="53b1c738-5a1a-47b3-b285-7b862c45e499" containerName="main" containerID="cri-o://e42cc379bdb0b20d3b8e41ff5369f51b9396d70f0d31c883e2b7eac8719190f4" gracePeriod=45
Sep 14 03:11:37.139997 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.139963    2580 generic.go:356] "Generic (PLEG): container finished" podID="53b1c738-5a1a-47b3-b285-7b862c45e499" containerID="e42cc379bdb0b20d3b8e41ff5369f51b9396d70f0d31c883e2b7eac8719190f4" exitCode=0
Sep 14 03:11:37.140182 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.140044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp" event={"ID":"53b1c738-5a1a-47b3-b285-7b862c45e499","Type":"ContainerDied","Data":"e42cc379bdb0b20d3b8e41ff5369f51b9396d70f0d31c883e2b7eac8719190f4"}
Sep 14 03:11:37.221945 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.221894    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:11:37.277398 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.277371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-tmp-dir\") pod \"53b1c738-5a1a-47b3-b285-7b862c45e499\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") "
Sep 14 03:11:37.277515 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.277459    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b1c738-5a1a-47b3-b285-7b862c45e499-tls-certs\") pod \"53b1c738-5a1a-47b3-b285-7b862c45e499\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") "
Sep 14 03:11:37.277515 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.277488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-model-cache\") pod \"53b1c738-5a1a-47b3-b285-7b862c45e499\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") "
Sep 14 03:11:37.277588 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.277555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57lsd\" (UniqueName: \"kubernetes.io/projected/53b1c738-5a1a-47b3-b285-7b862c45e499-kube-api-access-57lsd\") pod \"53b1c738-5a1a-47b3-b285-7b862c45e499\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") "
Sep 14 03:11:37.277627 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.277619    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-home\") pod \"53b1c738-5a1a-47b3-b285-7b862c45e499\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") "
Sep 14 03:11:37.277665 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.277633    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53b1c738-5a1a-47b3-b285-7b862c45e499" (UID: "53b1c738-5a1a-47b3-b285-7b862c45e499"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:11:37.277711 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.277680    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-dshm\") pod \"53b1c738-5a1a-47b3-b285-7b862c45e499\" (UID: \"53b1c738-5a1a-47b3-b285-7b862c45e499\") "
Sep 14 03:11:37.277833 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.277805    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-model-cache" (OuterVolumeSpecName: "model-cache") pod "53b1c738-5a1a-47b3-b285-7b862c45e499" (UID: "53b1c738-5a1a-47b3-b285-7b862c45e499"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:11:37.277833 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.277829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-home" (OuterVolumeSpecName: "home") pod "53b1c738-5a1a-47b3-b285-7b862c45e499" (UID: "53b1c738-5a1a-47b3-b285-7b862c45e499"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:11:37.278070 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.278024    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:11:37.278070 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.278069    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:11:37.278238 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.278083    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:11:37.279686 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.279661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b1c738-5a1a-47b3-b285-7b862c45e499-kube-api-access-57lsd" (OuterVolumeSpecName: "kube-api-access-57lsd") pod "53b1c738-5a1a-47b3-b285-7b862c45e499" (UID: "53b1c738-5a1a-47b3-b285-7b862c45e499"). InnerVolumeSpecName "kube-api-access-57lsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:11:37.279763 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.279682    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-dshm" (OuterVolumeSpecName: "dshm") pod "53b1c738-5a1a-47b3-b285-7b862c45e499" (UID: "53b1c738-5a1a-47b3-b285-7b862c45e499"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:11:37.279763 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.279682    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b1c738-5a1a-47b3-b285-7b862c45e499-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53b1c738-5a1a-47b3-b285-7b862c45e499" (UID: "53b1c738-5a1a-47b3-b285-7b862c45e499"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:11:37.379132 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.379106    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b1c738-5a1a-47b3-b285-7b862c45e499-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:11:37.379132 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.379128    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57lsd\" (UniqueName: \"kubernetes.io/projected/53b1c738-5a1a-47b3-b285-7b862c45e499-kube-api-access-57lsd\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:11:37.379263 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:37.379137    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b1c738-5a1a-47b3-b285-7b862c45e499-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:11:38.146934 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:38.146854    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"
Sep 14 03:11:38.146934 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:38.146866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp" event={"ID":"53b1c738-5a1a-47b3-b285-7b862c45e499","Type":"ContainerDied","Data":"b05527214d98408c3e073e6f953962b5a541aad5c8578f9637cf8524a6b7b484"}
Sep 14 03:11:38.146934 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:38.146916    2580 scope.go:117] "RemoveContainer" containerID="e42cc379bdb0b20d3b8e41ff5369f51b9396d70f0d31c883e2b7eac8719190f4"
Sep 14 03:11:38.163349 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:38.163325    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"]
Sep 14 03:11:38.167760 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:38.167736    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f002a1/test-model-f002a1-kserve-bc9954745-9qdrp"]
Sep 14 03:11:39.757583 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:39.757549    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53b1c738-5a1a-47b3-b285-7b862c45e499" path="/var/lib/kubelet/pods/53b1c738-5a1a-47b3-b285-7b862c45e499/volumes"
Sep 14 03:11:46.048759 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.048730    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"]
Sep 14 03:11:46.102523 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.102492    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"]
Sep 14 03:11:46.102873 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.102859    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b1c738-5a1a-47b3-b285-7b862c45e499" containerName="main"
Sep 14 03:11:46.102873 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.102873    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b1c738-5a1a-47b3-b285-7b862c45e499" containerName="main"
Sep 14 03:11:46.102996 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.102894    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="763ec406-383b-458d-af55-2b45a337efa5" containerName="payload-pre-processing"
Sep 14 03:11:46.102996 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.102899    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="763ec406-383b-458d-af55-2b45a337efa5" containerName="payload-pre-processing"
Sep 14 03:11:46.102996 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.102965    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b1c738-5a1a-47b3-b285-7b862c45e499" containerName="main"
Sep 14 03:11:46.102996 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.102974    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="763ec406-383b-458d-af55-2b45a337efa5" containerName="payload-pre-processing"
Sep 14 03:11:46.106174 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.106156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:11:46.110296 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.110279    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-12ecd131-dockercfg-5l6lp\""
Sep 14 03:11:46.110625 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.110611    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-12ecd131\""
Sep 14 03:11:46.120560 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.120541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"]
Sep 14 03:11:46.158871 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.158845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067f808f-2dc5-498d-a909-d184bbb7968c-plugins-config-volume\") pod \"payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w\" (UID: \"067f808f-2dc5-498d-a909-d184bbb7968c\") " pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:11:46.158968 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.158876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdrv\" (UniqueName: \"kubernetes.io/projected/067f808f-2dc5-498d-a909-d184bbb7968c-kube-api-access-7wdrv\") pod \"payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w\" (UID: \"067f808f-2dc5-498d-a909-d184bbb7968c\") " pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:11:46.259686 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.259655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067f808f-2dc5-498d-a909-d184bbb7968c-plugins-config-volume\") pod \"payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w\" (UID: \"067f808f-2dc5-498d-a909-d184bbb7968c\") " pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:11:46.259686 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.259692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdrv\" (UniqueName: \"kubernetes.io/projected/067f808f-2dc5-498d-a909-d184bbb7968c-kube-api-access-7wdrv\") pod \"payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w\" (UID: \"067f808f-2dc5-498d-a909-d184bbb7968c\") " pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:11:46.260368 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.260347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067f808f-2dc5-498d-a909-d184bbb7968c-plugins-config-volume\") pod \"payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w\" (UID: \"067f808f-2dc5-498d-a909-d184bbb7968c\") " pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:11:46.267759 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.267730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdrv\" (UniqueName: \"kubernetes.io/projected/067f808f-2dc5-498d-a909-d184bbb7968c-kube-api-access-7wdrv\") pod \"payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w\" (UID: \"067f808f-2dc5-498d-a909-d184bbb7968c\") " pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:11:46.415890 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.415823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:11:46.689568 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:46.689541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"]
Sep 14 03:11:46.690948 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:11:46.690920    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067f808f_2dc5_498d_a909_d184bbb7968c.slice/crio-08ecb180930d8c886a4af7a4b6e90fcea47fe8fde3995df538ad03e355daba85 WatchSource:0}: Error finding container 08ecb180930d8c886a4af7a4b6e90fcea47fe8fde3995df538ad03e355daba85: Status 404 returned error can't find the container with id 08ecb180930d8c886a4af7a4b6e90fcea47fe8fde3995df538ad03e355daba85
Sep 14 03:11:47.208354 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:47.208314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w" event={"ID":"067f808f-2dc5-498d-a909-d184bbb7968c","Type":"ContainerStarted","Data":"914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c"}
Sep 14 03:11:47.208810 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:47.208380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w" event={"ID":"067f808f-2dc5-498d-a909-d184bbb7968c","Type":"ContainerStarted","Data":"08ecb180930d8c886a4af7a4b6e90fcea47fe8fde3995df538ad03e355daba85"}
Sep 14 03:11:47.208810 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:47.208517    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:11:47.232179 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:47.232128    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w" podStartSLOduration=1.232107633 podStartE2EDuration="1.232107633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:47.230791456 +0000 UTC m=+1173.956749507" watchObservedRunningTime="2026-09-14 03:11:47.232107633 +0000 UTC m=+1173.958065687"
Sep 14 03:11:58.216061 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:11:58.216005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:12:00.612079 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:00.612041    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:12:01.067599 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.067544    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz" podUID="6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" containerName="main" containerID="cri-o://ce7eb9afdc68ec3524c2e5d11a5b3dfb19fbc0af5d48f4a72bba57fda1119f92" gracePeriod=45
Sep 14 03:12:01.293305 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.293272    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" containerID="ce7eb9afdc68ec3524c2e5d11a5b3dfb19fbc0af5d48f4a72bba57fda1119f92" exitCode=0
Sep 14 03:12:01.293465 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.293316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz" event={"ID":"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf","Type":"ContainerDied","Data":"ce7eb9afdc68ec3524c2e5d11a5b3dfb19fbc0af5d48f4a72bba57fda1119f92"}
Sep 14 03:12:01.323517 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.323467    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:12:01.398740 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.398712    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tls-certs\") pod \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") "
Sep 14 03:12:01.398905 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.398851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-home\") pod \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") "
Sep 14 03:12:01.398905 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.398888    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tmp-dir\") pod \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") "
Sep 14 03:12:01.399021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.398910    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwb8r\" (UniqueName: \"kubernetes.io/projected/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-kube-api-access-jwb8r\") pod \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") "
Sep 14 03:12:01.399021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.398942    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-model-cache\") pod \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") "
Sep 14 03:12:01.399021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.398990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-dshm\") pod \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\" (UID: \"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf\") "
Sep 14 03:12:01.399200 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.399169    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-home" (OuterVolumeSpecName: "home") pod "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" (UID: "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:01.399257 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.399185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" (UID: "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:01.399307 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.399290    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" (UID: "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:01.399548 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.399526    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:01.399632 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.399552    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:01.399632 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.399567    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:01.401117 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.401090    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" (UID: "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:12:01.401228 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.401170    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-dshm" (OuterVolumeSpecName: "dshm") pod "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" (UID: "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:01.401228 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.401214    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-kube-api-access-jwb8r" (OuterVolumeSpecName: "kube-api-access-jwb8r") pod "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" (UID: "6f116f5b-cbb8-4a8f-985c-f32c7f7defbf"). InnerVolumeSpecName "kube-api-access-jwb8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:12:01.500484 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.500456    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:01.500484 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.500481    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwb8r\" (UniqueName: \"kubernetes.io/projected/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-kube-api-access-jwb8r\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:01.500484 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:01.500489    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:02.300204 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:02.300172    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"
Sep 14 03:12:02.300204 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:02.300184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz" event={"ID":"6f116f5b-cbb8-4a8f-985c-f32c7f7defbf","Type":"ContainerDied","Data":"c811740466cdc9ef91b9c167b3102c83ecab7d4a26e6daef255220681701a8be"}
Sep 14 03:12:02.300690 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:02.300240    2580 scope.go:117] "RemoveContainer" containerID="ce7eb9afdc68ec3524c2e5d11a5b3dfb19fbc0af5d48f4a72bba57fda1119f92"
Sep 14 03:12:02.321748 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:02.321718    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"]
Sep 14 03:12:02.326827 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:02.326803    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4eb26b/test-model-4eb26b-kserve-5b5b79589f-twtrz"]
Sep 14 03:12:03.757450 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:03.757419    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" path="/var/lib/kubelet/pods/6f116f5b-cbb8-4a8f-985c-f32c7f7defbf/volumes"
Sep 14 03:12:05.824582 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:05.824543    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"]
Sep 14 03:12:05.825019 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:05.824779    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy" containerID="cri-o://5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a" gracePeriod=30
Sep 14 03:12:08.361280 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.361252    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"]
Sep 14 03:12:08.361701 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.361687    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" containerName="main"
Sep 14 03:12:08.361748 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.361703    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" containerName="main"
Sep 14 03:12:08.361784 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.361767    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f116f5b-cbb8-4a8f-985c-f32c7f7defbf" containerName="main"
Sep 14 03:12:08.366083 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.366023    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.368247 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.368228    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a752f\"/\"distinct-2-w0-3a752f-kserve-self-signed-certs\""
Sep 14 03:12:08.376249 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.376226    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"]
Sep 14 03:12:08.464590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.464561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tls-certs\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.464756 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.464606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-model-cache\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.464756 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.464688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-dshm\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.464881 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.464774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4fd\" (UniqueName: \"kubernetes.io/projected/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-kube-api-access-wc4fd\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.464881 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.464827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-home\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.464881 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.464856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tmp-dir\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.565958 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.565919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-home\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.565958 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.565960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tmp-dir\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.566248 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.566064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tls-certs\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.566248 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.566117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-model-cache\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.566248 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.566167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-dshm\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.566248 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.566236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4fd\" (UniqueName: \"kubernetes.io/projected/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-kube-api-access-wc4fd\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.566460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.566328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-home\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.566460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.566381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tmp-dir\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.566460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.566434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-model-cache\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.568355 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.568326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-dshm\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.568506 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.568489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tls-certs\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.583059 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.583024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4fd\" (UniqueName: \"kubernetes.io/projected/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-kube-api-access-wc4fd\") pod \"distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") " pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.676133 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.676055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:08.952672 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:08.952645    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"]
Sep 14 03:12:08.954387 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:12:08.954363    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9e9616a_a789_4dcf_b4fc_4b6c4b64e06b.slice/crio-a96449725c76177b3588d7e95fe89d1e8137c1b9bcfdf044f64cc8d7d3f11a26 WatchSource:0}: Error finding container a96449725c76177b3588d7e95fe89d1e8137c1b9bcfdf044f64cc8d7d3f11a26: Status 404 returned error can't find the container with id a96449725c76177b3588d7e95fe89d1e8137c1b9bcfdf044f64cc8d7d3f11a26
Sep 14 03:12:09.346662 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:09.346628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5" event={"ID":"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b","Type":"ContainerStarted","Data":"ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e"}
Sep 14 03:12:09.346825 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:09.346671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5" event={"ID":"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b","Type":"ContainerStarted","Data":"a96449725c76177b3588d7e95fe89d1e8137c1b9bcfdf044f64cc8d7d3f11a26"}
Sep 14 03:12:09.369088 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:09.369026    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5" podStartSLOduration=1.369011668 podStartE2EDuration="1.369011668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:12:09.367448142 +0000 UTC m=+1196.093406194" watchObservedRunningTime="2026-09-14 03:12:09.369011668 +0000 UTC m=+1196.094969719"
Sep 14 03:12:11.086509 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.086445    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:12:11.091760 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.091738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-istio-token\") pod \"d26b794a-4741-47f9-9773-68b2731d4bee\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") "
Sep 14 03:12:11.091899 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.091771    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26b794a-4741-47f9-9773-68b2731d4bee-istiod-ca-cert\") pod \"d26b794a-4741-47f9-9773-68b2731d4bee\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") "
Sep 14 03:12:11.091899 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.091812    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-socket\") pod \"d26b794a-4741-47f9-9773-68b2731d4bee\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") "
Sep 14 03:12:11.091899 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.091851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqmw2\" (UniqueName: \"kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-kube-api-access-xqmw2\") pod \"d26b794a-4741-47f9-9773-68b2731d4bee\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") "
Sep 14 03:12:11.091899 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.091886    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-credential-socket\") pod \"d26b794a-4741-47f9-9773-68b2731d4bee\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") "
Sep 14 03:12:11.092136 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.091912    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-data\") pod \"d26b794a-4741-47f9-9773-68b2731d4bee\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") "
Sep 14 03:12:11.092136 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.091939    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-certs\") pod \"d26b794a-4741-47f9-9773-68b2731d4bee\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") "
Sep 14 03:12:11.092136 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.092009    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-envoy\") pod \"d26b794a-4741-47f9-9773-68b2731d4bee\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") "
Sep 14 03:12:11.092136 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.092059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26b794a-4741-47f9-9773-68b2731d4bee-istio-podinfo\") pod \"d26b794a-4741-47f9-9773-68b2731d4bee\" (UID: \"d26b794a-4741-47f9-9773-68b2731d4bee\") "
Sep 14 03:12:11.092351 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.092156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d26b794a-4741-47f9-9773-68b2731d4bee" (UID: "d26b794a-4741-47f9-9773-68b2731d4bee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:11.092351 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.092155    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26b794a-4741-47f9-9773-68b2731d4bee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d26b794a-4741-47f9-9773-68b2731d4bee" (UID: "d26b794a-4741-47f9-9773-68b2731d4bee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:12:11.092462 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.092272    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-data" (OuterVolumeSpecName: "istio-data") pod "d26b794a-4741-47f9-9773-68b2731d4bee" (UID: "d26b794a-4741-47f9-9773-68b2731d4bee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:11.092462 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.092359    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d26b794a-4741-47f9-9773-68b2731d4bee" (UID: "d26b794a-4741-47f9-9773-68b2731d4bee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:11.092462 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.092354    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d26b794a-4741-47f9-9773-68b2731d4bee" (UID: "d26b794a-4741-47f9-9773-68b2731d4bee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:11.092616 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.092469    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:11.092616 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.092492    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26b794a-4741-47f9-9773-68b2731d4bee-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:11.094014 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.093990    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-istio-token" (OuterVolumeSpecName: "istio-token") pod "d26b794a-4741-47f9-9773-68b2731d4bee" (UID: "d26b794a-4741-47f9-9773-68b2731d4bee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:12:11.094200 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.094176    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-kube-api-access-xqmw2" (OuterVolumeSpecName: "kube-api-access-xqmw2") pod "d26b794a-4741-47f9-9773-68b2731d4bee" (UID: "d26b794a-4741-47f9-9773-68b2731d4bee"). InnerVolumeSpecName "kube-api-access-xqmw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:12:11.094273 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.094223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d26b794a-4741-47f9-9773-68b2731d4bee" (UID: "d26b794a-4741-47f9-9773-68b2731d4bee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:11.094425 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.094400    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d26b794a-4741-47f9-9773-68b2731d4bee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d26b794a-4741-47f9-9773-68b2731d4bee" (UID: "d26b794a-4741-47f9-9773-68b2731d4bee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 03:12:11.193828 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.193800    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqmw2\" (UniqueName: \"kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-kube-api-access-xqmw2\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:11.193828 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.193824    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:11.193828 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.193834    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:11.194076 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.193843    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:11.194076 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.193853    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26b794a-4741-47f9-9773-68b2731d4bee-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:11.194076 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.193861    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26b794a-4741-47f9-9773-68b2731d4bee-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:11.194076 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.193869    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26b794a-4741-47f9-9773-68b2731d4bee-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:12:11.360251 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.360160    2580 generic.go:356] "Generic (PLEG): container finished" podID="d26b794a-4741-47f9-9773-68b2731d4bee" containerID="5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a" exitCode=0
Sep 14 03:12:11.360251 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.360211    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"
Sep 14 03:12:11.360431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.360259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" event={"ID":"d26b794a-4741-47f9-9773-68b2731d4bee","Type":"ContainerDied","Data":"5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a"}
Sep 14 03:12:11.360431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.360312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp" event={"ID":"d26b794a-4741-47f9-9773-68b2731d4bee","Type":"ContainerDied","Data":"9f8a4f80d59c69ba6861d4dc2c3c20d14ac2c464423edcc1983e64cfde13c9ad"}
Sep 14 03:12:11.360431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.360340    2580 scope.go:117] "RemoveContainer" containerID="5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a"
Sep 14 03:12:11.377898 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.377871    2580 scope.go:117] "RemoveContainer" containerID="182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a"
Sep 14 03:12:11.386250 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.386232    2580 scope.go:117] "RemoveContainer" containerID="5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a"
Sep 14 03:12:11.386513 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:12:11.386492    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a\": container with ID starting with 5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a not found: ID does not exist" containerID="5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a"
Sep 14 03:12:11.386605 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.386519    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a"} err="failed to get container status \"5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a\": rpc error: code = NotFound desc = could not find container \"5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a\": container with ID starting with 5ebf15ad311bf9ad0a4f1e6b135880995132413e8c2abb62554efc9e2f7c9a5a not found: ID does not exist"
Sep 14 03:12:11.386605 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.386536    2580 scope.go:117] "RemoveContainer" containerID="182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a"
Sep 14 03:12:11.386778 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:12:11.386763    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a\": container with ID starting with 182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a not found: ID does not exist" containerID="182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a"
Sep 14 03:12:11.386813 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.386782    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a"} err="failed to get container status \"182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a\": rpc error: code = NotFound desc = could not find container \"182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a\": container with ID starting with 182e69ea13d1cba653890b40a344a35496765c2faa4457cf6969590a7a987a5a not found: ID does not exist"
Sep 14 03:12:11.391112 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.391090    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"]
Sep 14 03:12:11.398710 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.398689    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f002a1-openshift-default-6b669985db-8v2pp"]
Sep 14 03:12:11.756975 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:11.756935    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" path="/var/lib/kubelet/pods/d26b794a-4741-47f9-9773-68b2731d4bee/volumes"
Sep 14 03:12:13.797121 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:13.797095    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:12:13.799054 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:13.799011    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:12:13.889233 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:13.889206    2580 scope.go:117] "RemoveContainer" containerID="8835d1f319a21d6b3275d860bc33a9ada0c50d33088eaf105486d7ad40905371"
Sep 14 03:12:18.677273 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:18.677239    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:18.677835 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:18.677286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:18.689856 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:18.689832    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:19.417534 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:19.417503    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:12:34.064562 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:12:34.064524    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:13:07.721694 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.721615    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"]
Sep 14 03:13:07.722155 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.722010    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy"
Sep 14 03:13:07.722155 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.722021    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy"
Sep 14 03:13:07.722155 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.722046    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy"
Sep 14 03:13:07.722155 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.722052    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy"
Sep 14 03:13:07.722155 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.722129    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy"
Sep 14 03:13:07.722155 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.722138    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26b794a-4741-47f9-9773-68b2731d4bee" containerName="istio-proxy"
Sep 14 03:13:07.725379 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.725362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:13:07.733382 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.733359    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-177593\""
Sep 14 03:13:07.739324 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.739301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"]
Sep 14 03:13:07.806418 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.806391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a957b12-750d-470c-8516-9b613f12f011-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9\" (UID: \"9a957b12-750d-470c-8516-9b613f12f011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:13:07.806566 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.806430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcr4h\" (UniqueName: \"kubernetes.io/projected/9a957b12-750d-470c-8516-9b613f12f011-kube-api-access-bcr4h\") pod \"payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9\" (UID: \"9a957b12-750d-470c-8516-9b613f12f011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:13:07.907632 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.907594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a957b12-750d-470c-8516-9b613f12f011-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9\" (UID: \"9a957b12-750d-470c-8516-9b613f12f011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:13:07.907818 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.907644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcr4h\" (UniqueName: \"kubernetes.io/projected/9a957b12-750d-470c-8516-9b613f12f011-kube-api-access-bcr4h\") pod \"payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9\" (UID: \"9a957b12-750d-470c-8516-9b613f12f011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:13:07.908325 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.908301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a957b12-750d-470c-8516-9b613f12f011-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9\" (UID: \"9a957b12-750d-470c-8516-9b613f12f011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:13:07.915810 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:07.915787    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcr4h\" (UniqueName: \"kubernetes.io/projected/9a957b12-750d-470c-8516-9b613f12f011-kube-api-access-bcr4h\") pod \"payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9\" (UID: \"9a957b12-750d-470c-8516-9b613f12f011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:13:08.035976 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:08.035946    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:13:08.255229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:08.255191    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:13:08.308590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:08.308561    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"]
Sep 14 03:13:08.308718 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:13:08.308653    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a957b12_750d_470c_8516_9b613f12f011.slice/crio-2b56efedbd80eb728a3c7f9a0a658569c10b13eec7e5280d89adb61613d97ca1 WatchSource:0}: Error finding container 2b56efedbd80eb728a3c7f9a0a658569c10b13eec7e5280d89adb61613d97ca1: Status 404 returned error can't find the container with id 2b56efedbd80eb728a3c7f9a0a658569c10b13eec7e5280d89adb61613d97ca1
Sep 14 03:13:08.687667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:08.687583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9" event={"ID":"9a957b12-750d-470c-8516-9b613f12f011","Type":"ContainerStarted","Data":"a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0"}
Sep 14 03:13:08.687667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:08.687630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9" event={"ID":"9a957b12-750d-470c-8516-9b613f12f011","Type":"ContainerStarted","Data":"2b56efedbd80eb728a3c7f9a0a658569c10b13eec7e5280d89adb61613d97ca1"}
Sep 14 03:13:08.687859 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:08.687678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:13:08.712454 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:08.712410    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9" podStartSLOduration=1.71239665 podStartE2EDuration="1.71239665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:13:08.710563589 +0000 UTC m=+1255.436521643" watchObservedRunningTime="2026-09-14 03:13:08.71239665 +0000 UTC m=+1255.438354740"
Sep 14 03:13:13.927258 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:13.927216    2580 scope.go:117] "RemoveContainer" containerID="0e27c5dc11c0034b5df7abf7247961150d5aa1cee199420247711e7f70f7fca5"
Sep 14 03:13:19.695961 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:19.695931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:13:40.859044 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:40.859004    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:13:51.972020 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:51.971986    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"]
Sep 14 03:13:51.975469 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:51.975453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:51.978695 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:51.978673    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b32aa\"/\"default-dockercfg-rjpdw\""
Sep 14 03:13:51.978695 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:51.978682    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b32aa\"/\"openshift-service-ca.crt\""
Sep 14 03:13:51.978882 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:51.978682    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b32aa\"/\"auth-test-model-2b32aa-kserve-self-signed-certs\""
Sep 14 03:13:51.978882 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:51.978714    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b32aa\"/\"kube-root-ca.crt\""
Sep 14 03:13:51.985006 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:51.984981    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"]
Sep 14 03:13:52.014055 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.014006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-home\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.014167 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.014074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-model-cache\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.014167 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.014124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-dshm\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.014246 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.014194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tmp-dir\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.014281 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.014252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tls-certs\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.014316 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.014280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sds6k\" (UniqueName: \"kubernetes.io/projected/80b5a27e-bbb5-4316-87b0-88803a9ffc21-kube-api-access-sds6k\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.114839 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.114803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tmp-dir\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.115013 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.114855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tls-certs\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.115013 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.114884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sds6k\" (UniqueName: \"kubernetes.io/projected/80b5a27e-bbb5-4316-87b0-88803a9ffc21-kube-api-access-sds6k\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.115013 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.114931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-home\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.115234 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.115078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-model-cache\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.115234 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.115139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-dshm\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.115234 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.115219    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tmp-dir\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.115383 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.115264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-home\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.115383 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.115375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-model-cache\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.117165 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.117148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-dshm\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.117467 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.117451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tls-certs\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.122946 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.122924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sds6k\" (UniqueName: \"kubernetes.io/projected/80b5a27e-bbb5-4316-87b0-88803a9ffc21-kube-api-access-sds6k\") pod \"auth-test-model-2b32aa-kserve-54f6f9967f-98whb\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.286307 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.286277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:13:52.560478 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.560296    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"]
Sep 14 03:13:52.562696 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:13:52.562671    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80b5a27e_bbb5_4316_87b0_88803a9ffc21.slice/crio-910b1ed83680cb2c775903c7ac59e5572ad224d315915c85948f5f1856dda58a WatchSource:0}: Error finding container 910b1ed83680cb2c775903c7ac59e5572ad224d315915c85948f5f1856dda58a: Status 404 returned error can't find the container with id 910b1ed83680cb2c775903c7ac59e5572ad224d315915c85948f5f1856dda58a
Sep 14 03:13:52.946984 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.946885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb" event={"ID":"80b5a27e-bbb5-4316-87b0-88803a9ffc21","Type":"ContainerStarted","Data":"cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05"}
Sep 14 03:13:52.946984 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.946940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb" event={"ID":"80b5a27e-bbb5-4316-87b0-88803a9ffc21","Type":"ContainerStarted","Data":"910b1ed83680cb2c775903c7ac59e5572ad224d315915c85948f5f1856dda58a"}
Sep 14 03:13:52.967276 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:13:52.967230    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb" podStartSLOduration=1.967210715 podStartE2EDuration="1.967210715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:13:52.96467066 +0000 UTC m=+1299.690628712" watchObservedRunningTime="2026-09-14 03:13:52.967210715 +0000 UTC m=+1299.693168768"
Sep 14 03:14:02.287332 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:02.287297    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:14:02.287735 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:02.287358    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:14:02.300753 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:02.300728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:14:02.893471 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:02.893438    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"]
Sep 14 03:14:02.893663 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:02.893638    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w" podUID="067f808f-2dc5-498d-a909-d184bbb7968c" containerName="payload-processing" containerID="cri-o://914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c" gracePeriod=30
Sep 14 03:14:03.013477 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:03.013450    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:14:03.157292 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:03.157220    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:14:03.215917 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:03.215884    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wdrv\" (UniqueName: \"kubernetes.io/projected/067f808f-2dc5-498d-a909-d184bbb7968c-kube-api-access-7wdrv\") pod \"067f808f-2dc5-498d-a909-d184bbb7968c\" (UID: \"067f808f-2dc5-498d-a909-d184bbb7968c\") "
Sep 14 03:14:03.216078 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:03.215945    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067f808f-2dc5-498d-a909-d184bbb7968c-plugins-config-volume\") pod \"067f808f-2dc5-498d-a909-d184bbb7968c\" (UID: \"067f808f-2dc5-498d-a909-d184bbb7968c\") "
Sep 14 03:14:03.216387 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:03.216362    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067f808f-2dc5-498d-a909-d184bbb7968c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "067f808f-2dc5-498d-a909-d184bbb7968c" (UID: "067f808f-2dc5-498d-a909-d184bbb7968c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:14:03.217968 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:03.217945    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067f808f-2dc5-498d-a909-d184bbb7968c-kube-api-access-7wdrv" (OuterVolumeSpecName: "kube-api-access-7wdrv") pod "067f808f-2dc5-498d-a909-d184bbb7968c" (UID: "067f808f-2dc5-498d-a909-d184bbb7968c"). InnerVolumeSpecName "kube-api-access-7wdrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:14:03.317700 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:03.317665    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wdrv\" (UniqueName: \"kubernetes.io/projected/067f808f-2dc5-498d-a909-d184bbb7968c-kube-api-access-7wdrv\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:14:03.317700 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:03.317695    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067f808f-2dc5-498d-a909-d184bbb7968c-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:14:04.008211 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:04.008181    2580 generic.go:356] "Generic (PLEG): container finished" podID="067f808f-2dc5-498d-a909-d184bbb7968c" containerID="914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c" exitCode=0
Sep 14 03:14:04.008376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:04.008223    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"
Sep 14 03:14:04.008376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:04.008264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w" event={"ID":"067f808f-2dc5-498d-a909-d184bbb7968c","Type":"ContainerDied","Data":"914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c"}
Sep 14 03:14:04.008376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:04.008310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w" event={"ID":"067f808f-2dc5-498d-a909-d184bbb7968c","Type":"ContainerDied","Data":"08ecb180930d8c886a4af7a4b6e90fcea47fe8fde3995df538ad03e355daba85"}
Sep 14 03:14:04.008376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:04.008334    2580 scope.go:117] "RemoveContainer" containerID="914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c"
Sep 14 03:14:04.017248 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:04.017231    2580 scope.go:117] "RemoveContainer" containerID="914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c"
Sep 14 03:14:04.017539 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:14:04.017515    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c\": container with ID starting with 914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c not found: ID does not exist" containerID="914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c"
Sep 14 03:14:04.017623 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:04.017551    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c"} err="failed to get container status \"914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c\": rpc error: code = NotFound desc = could not find container \"914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c\": container with ID starting with 914978a54d8abe6d8f4a0cb2554749db72796a667516f052e5263c587ba74c7c not found: ID does not exist"
Sep 14 03:14:04.034943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:04.034918    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"]
Sep 14 03:14:04.042391 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:04.042371    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12ecd131-84b7fdbdc7-qtf4w"]
Sep 14 03:14:05.757871 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:05.757842    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="067f808f-2dc5-498d-a909-d184bbb7968c" path="/var/lib/kubelet/pods/067f808f-2dc5-498d-a909-d184bbb7968c/volumes"
Sep 14 03:14:10.757972 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:10.757937    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:14:42.255936 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:14:42.255857    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:15:00.176996 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.176957    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg"]
Sep 14 03:15:00.177582 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.177563    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067f808f-2dc5-498d-a909-d184bbb7968c" containerName="payload-processing"
Sep 14 03:15:00.177627 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.177586    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="067f808f-2dc5-498d-a909-d184bbb7968c" containerName="payload-processing"
Sep 14 03:15:00.177711 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.177699    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="067f808f-2dc5-498d-a909-d184bbb7968c" containerName="payload-processing"
Sep 14 03:15:00.181338 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.181322    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq"]
Sep 14 03:15:00.181455 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.181437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg"
Sep 14 03:15:00.184741 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.184724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq"
Sep 14 03:15:00.191951 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.191928    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sv5sd\""
Sep 14 03:15:00.192081 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.191994    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg"
Sep 14 03:15:00.200225 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.200205    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq"
Sep 14 03:15:00.231812 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.231704    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg"]
Sep 14 03:15:00.248268 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.248241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq"]
Sep 14 03:15:00.481588 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.481558    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg"]
Sep 14 03:15:00.483201 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:15:00.483166    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ff5e95_0418_4412_8c1d_dd5f6a6bb5bb.slice/crio-2a14ebac4fb0bcee9f8fc28bce261eeea692e04a80a98bd8961c690c87a35898 WatchSource:0}: Error finding container 2a14ebac4fb0bcee9f8fc28bce261eeea692e04a80a98bd8961c690c87a35898: Status 404 returned error can't find the container with id 2a14ebac4fb0bcee9f8fc28bce261eeea692e04a80a98bd8961c690c87a35898
Sep 14 03:15:00.494107 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:00.494088    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq"]
Sep 14 03:15:00.497923 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:15:00.497902    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33339fca_0aef_40d4_88de_7c717c527dd1.slice/crio-cc93d8aedb093aecd9588b287b263bcb57f2b06922dd6cf2737e9396f1c2b0cd WatchSource:0}: Error finding container cc93d8aedb093aecd9588b287b263bcb57f2b06922dd6cf2737e9396f1c2b0cd: Status 404 returned error can't find the container with id cc93d8aedb093aecd9588b287b263bcb57f2b06922dd6cf2737e9396f1c2b0cd
Sep 14 03:15:01.349194 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:01.349148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg" event={"ID":"24ff5e95-0418-4412-8c1d-dd5f6a6bb5bb","Type":"ContainerStarted","Data":"fb27247e91617c22415fa85d7c3059950f17730354d1b47f4a2a6e379f2add1d"}
Sep 14 03:15:01.349639 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:01.349210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg" event={"ID":"24ff5e95-0418-4412-8c1d-dd5f6a6bb5bb","Type":"ContainerStarted","Data":"2a14ebac4fb0bcee9f8fc28bce261eeea692e04a80a98bd8961c690c87a35898"}
Sep 14 03:15:01.350984 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:01.350954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq" event={"ID":"33339fca-0aef-40d4-88de-7c717c527dd1","Type":"ContainerStarted","Data":"2b42943e041d868b685a7d45aec07f844b6fa4e4efae7dc38bce20a540506ce4"}
Sep 14 03:15:01.351135 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:01.351020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq" event={"ID":"33339fca-0aef-40d4-88de-7c717c527dd1","Type":"ContainerStarted","Data":"cc93d8aedb093aecd9588b287b263bcb57f2b06922dd6cf2737e9396f1c2b0cd"}
Sep 14 03:15:01.364592 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:01.364551    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg" podStartSLOduration=1.364536889 podStartE2EDuration="1.364536889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:01.363715887 +0000 UTC m=+1368.089673939" watchObservedRunningTime="2026-09-14 03:15:01.364536889 +0000 UTC m=+1368.090494991"
Sep 14 03:15:01.380201 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:01.380158    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq" podStartSLOduration=1.380143895 podStartE2EDuration="1.380143895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:01.378324633 +0000 UTC m=+1368.104282686" watchObservedRunningTime="2026-09-14 03:15:01.380143895 +0000 UTC m=+1368.106101947"
Sep 14 03:15:02.358452 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:02.358417    2580 generic.go:356] "Generic (PLEG): container finished" podID="33339fca-0aef-40d4-88de-7c717c527dd1" containerID="2b42943e041d868b685a7d45aec07f844b6fa4e4efae7dc38bce20a540506ce4" exitCode=0
Sep 14 03:15:02.358825 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:02.358505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq" event={"ID":"33339fca-0aef-40d4-88de-7c717c527dd1","Type":"ContainerDied","Data":"2b42943e041d868b685a7d45aec07f844b6fa4e4efae7dc38bce20a540506ce4"}
Sep 14 03:15:02.360166 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:02.360143    2580 generic.go:356] "Generic (PLEG): container finished" podID="24ff5e95-0418-4412-8c1d-dd5f6a6bb5bb" containerID="fb27247e91617c22415fa85d7c3059950f17730354d1b47f4a2a6e379f2add1d" exitCode=0
Sep 14 03:15:02.360298 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:02.360179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg" event={"ID":"24ff5e95-0418-4412-8c1d-dd5f6a6bb5bb","Type":"ContainerDied","Data":"fb27247e91617c22415fa85d7c3059950f17730354d1b47f4a2a6e379f2add1d"}
Sep 14 03:15:03.531135 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:03.531111    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq"
Sep 14 03:15:03.534079 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:03.534059    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg"
Sep 14 03:15:04.372971 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:04.372941    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg"
Sep 14 03:15:04.372971 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:04.372960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg" event={"ID":"24ff5e95-0418-4412-8c1d-dd5f6a6bb5bb","Type":"ContainerDied","Data":"2a14ebac4fb0bcee9f8fc28bce261eeea692e04a80a98bd8961c690c87a35898"}
Sep 14 03:15:04.373200 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:04.373007    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a14ebac4fb0bcee9f8fc28bce261eeea692e04a80a98bd8961c690c87a35898"
Sep 14 03:15:04.374362 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:04.374341    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq"
Sep 14 03:15:04.374464 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:04.374336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq" event={"ID":"33339fca-0aef-40d4-88de-7c717c527dd1","Type":"ContainerDied","Data":"cc93d8aedb093aecd9588b287b263bcb57f2b06922dd6cf2737e9396f1c2b0cd"}
Sep 14 03:15:04.374504 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:04.374464    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc93d8aedb093aecd9588b287b263bcb57f2b06922dd6cf2737e9396f1c2b0cd"
Sep 14 03:15:18.870301 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.870266    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"]
Sep 14 03:15:18.870833 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.870812    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ff5e95-0418-4412-8c1d-dd5f6a6bb5bb" containerName="cleanup"
Sep 14 03:15:18.870922 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.870836    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ff5e95-0418-4412-8c1d-dd5f6a6bb5bb" containerName="cleanup"
Sep 14 03:15:18.870922 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.870859    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33339fca-0aef-40d4-88de-7c717c527dd1" containerName="cleanup"
Sep 14 03:15:18.870922 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.870868    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33339fca-0aef-40d4-88de-7c717c527dd1" containerName="cleanup"
Sep 14 03:15:18.871090 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.871010    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ff5e95-0418-4412-8c1d-dd5f6a6bb5bb" containerName="cleanup"
Sep 14 03:15:18.871090 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.871042    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="33339fca-0aef-40d4-88de-7c717c527dd1" containerName="cleanup"
Sep 14 03:15:18.873630 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.873610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:18.876814 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.876793    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"openshift-service-ca.crt\""
Sep 14 03:15:18.877367 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.877348    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"kube-root-ca.crt\""
Sep 14 03:15:18.877416 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.877364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"auth-test-model-177593-kserve-self-signed-certs\""
Sep 14 03:15:18.877416 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.877390    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"default-dockercfg-54rpf\""
Sep 14 03:15:18.889255 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:18.889231    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"]
Sep 14 03:15:19.000590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.000554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kg7j\" (UniqueName: \"kubernetes.io/projected/913fed4a-1b2d-49a7-98c6-f0bb4863308f-kube-api-access-4kg7j\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.000760 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.000683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-dshm\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.000760 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.000754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-home\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.000836 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.000787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tmp-dir\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.000836 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.000812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tls-certs\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.000911 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.000857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-model-cache\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.101812 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.101784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kg7j\" (UniqueName: \"kubernetes.io/projected/913fed4a-1b2d-49a7-98c6-f0bb4863308f-kube-api-access-4kg7j\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.101942 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.101843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-dshm\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.101942 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.101876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-home\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.101942 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.101900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tmp-dir\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.101942 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.101920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tls-certs\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.102146 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.102128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-model-cache\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.102386 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.102366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tmp-dir\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.102463 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.102395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-home\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.102463 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.102440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-model-cache\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.104330 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.104313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-dshm\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.104492 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.104476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tls-certs\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.109608 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.109583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kg7j\" (UniqueName: \"kubernetes.io/projected/913fed4a-1b2d-49a7-98c6-f0bb4863308f-kube-api-access-4kg7j\") pod \"auth-test-model-177593-kserve-5d95c697f9-w5x5g\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") " pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.183631 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.183582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:19.470514 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:15:19.470485    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod913fed4a_1b2d_49a7_98c6_f0bb4863308f.slice/crio-bececed80b95696ba73080e869f04a8ef1536b0c2db8c8aabd8e3d143c5ce060 WatchSource:0}: Error finding container bececed80b95696ba73080e869f04a8ef1536b0c2db8c8aabd8e3d143c5ce060: Status 404 returned error can't find the container with id bececed80b95696ba73080e869f04a8ef1536b0c2db8c8aabd8e3d143c5ce060
Sep 14 03:15:19.470636 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:19.470615    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"]
Sep 14 03:15:20.467852 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:20.467820    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:15:20.474160 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:20.474131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g" event={"ID":"913fed4a-1b2d-49a7-98c6-f0bb4863308f","Type":"ContainerStarted","Data":"19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb"}
Sep 14 03:15:20.474301 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:20.474170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g" event={"ID":"913fed4a-1b2d-49a7-98c6-f0bb4863308f","Type":"ContainerStarted","Data":"bececed80b95696ba73080e869f04a8ef1536b0c2db8c8aabd8e3d143c5ce060"}
Sep 14 03:15:20.491449 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:20.491407    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g" podStartSLOduration=2.491392553 podStartE2EDuration="2.491392553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:20.489620383 +0000 UTC m=+1387.215578438" watchObservedRunningTime="2026-09-14 03:15:20.491392553 +0000 UTC m=+1387.217350602"
Sep 14 03:15:29.183968 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:29.183935    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:29.184474 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:29.183996    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:29.196622 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:29.196598    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:29.536552 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:29.536521    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:15:56.661608 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:15:56.661572    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:16:13.971062 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:13.970976    2580 scope.go:117] "RemoveContainer" containerID="51d9f0a4d2890cb0049bb220db4316859db1abe1c96884e535b6fc1b82c9ced2"
Sep 14 03:16:13.979893 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:13.979868    2580 scope.go:117] "RemoveContainer" containerID="3c6fe9ad9aa3393d283cd59fb13fd14fa9b3bbfd1a6b2e22b73d5f010653a207"
Sep 14 03:16:23.703356 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.703323    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"]
Sep 14 03:16:23.708323 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.708302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.710617 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.710591    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-97da78-openshift-default-dockercfg-nr85h\""
Sep 14 03:16:23.724420 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.724396    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"]
Sep 14 03:16:23.800263 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.800221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-envoy\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.800458 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.800277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-podinfo\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.800458 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.800444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-credential-socket\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.800601 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.800483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-certs\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.800601 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.800546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-socket\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.800703 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.800642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgxxp\" (UniqueName: \"kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-kube-api-access-kgxxp\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.800761 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.800706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5abadfa-a761-43fe-afe6-9ab492992fb9-istiod-ca-cert\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.800761 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.800744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-data\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.800856 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.800766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-token\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902270 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5abadfa-a761-43fe-afe6-9ab492992fb9-istiod-ca-cert\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902270 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-data\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902457 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-token\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902457 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-envoy\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902457 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-podinfo\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902457 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-credential-socket\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902457 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-certs\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902716 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-socket\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902716 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgxxp\" (UniqueName: \"kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-kube-api-access-kgxxp\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902716 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-data\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902914 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-certs\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.902987 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.902949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-socket\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.903137 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.903116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5abadfa-a761-43fe-afe6-9ab492992fb9-istiod-ca-cert\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.903203 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.903127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-credential-socket\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.904735 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.904714    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-envoy\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.904991 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.904974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-podinfo\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.910257 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.910236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgxxp\" (UniqueName: \"kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-kube-api-access-kgxxp\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:23.910337 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:23.910266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-token\") pod \"e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") " pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:24.018495 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:24.018460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:24.291914 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:24.291875    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"]
Sep 14 03:16:24.292753 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:16:24.292727    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5abadfa_a761_43fe_afe6_9ab492992fb9.slice/crio-9f52c02bc8adb7c28715a19b732ced8a94155f36483caceb7bde05127f12598d WatchSource:0}: Error finding container 9f52c02bc8adb7c28715a19b732ced8a94155f36483caceb7bde05127f12598d: Status 404 returned error can't find the container with id 9f52c02bc8adb7c28715a19b732ced8a94155f36483caceb7bde05127f12598d
Sep 14 03:16:24.295449 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:24.295408    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:16:24.295534 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:24.295492    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:16:24.295580 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:24.295534    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:16:24.858731 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:24.858689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" event={"ID":"b5abadfa-a761-43fe-afe6-9ab492992fb9","Type":"ContainerStarted","Data":"c5f72ef5f3d03cd88e29f85b34e1c3ee8aa759e72333a71d3ff3a50001ba0d1f"}
Sep 14 03:16:24.859209 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:24.858749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" event={"ID":"b5abadfa-a761-43fe-afe6-9ab492992fb9","Type":"ContainerStarted","Data":"9f52c02bc8adb7c28715a19b732ced8a94155f36483caceb7bde05127f12598d"}
Sep 14 03:16:24.877645 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:24.877600    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" podStartSLOduration=1.877585559 podStartE2EDuration="1.877585559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:16:24.87614576 +0000 UTC m=+1451.602103811" watchObservedRunningTime="2026-09-14 03:16:24.877585559 +0000 UTC m=+1451.603543610"
Sep 14 03:16:25.018998 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:25.018967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:25.024171 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:25.024145    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:25.866699 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:25.866668    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:25.867587 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:25.867564    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:16:31.324932 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:31.324896    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"]
Sep 14 03:16:31.332142 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:31.332115    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"]
Sep 14 03:16:31.360267 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:31.360237    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"]
Sep 14 03:16:31.365166 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:31.365138    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"]
Sep 14 03:16:34.197937 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.197905    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"]
Sep 14 03:16:34.201736 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.201716    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:16:34.204201 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.204181    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-97da78\""
Sep 14 03:16:34.204464 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.204448    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-97da78-dockercfg-2x8xw\""
Sep 14 03:16:34.218899 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.218875    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"]
Sep 14 03:16:34.305773 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.305737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-plugins-config-volume\") pod \"payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d\" (UID: \"1d873fa5-6e7b-4b32-81b7-35a06abc2d81\") " pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:16:34.305935 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.305817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdtc5\" (UniqueName: \"kubernetes.io/projected/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-kube-api-access-wdtc5\") pod \"payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d\" (UID: \"1d873fa5-6e7b-4b32-81b7-35a06abc2d81\") " pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:16:34.406520 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.406482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-plugins-config-volume\") pod \"payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d\" (UID: \"1d873fa5-6e7b-4b32-81b7-35a06abc2d81\") " pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:16:34.406719 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.406553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdtc5\" (UniqueName: \"kubernetes.io/projected/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-kube-api-access-wdtc5\") pod \"payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d\" (UID: \"1d873fa5-6e7b-4b32-81b7-35a06abc2d81\") " pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:16:34.407229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.407209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-plugins-config-volume\") pod \"payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d\" (UID: \"1d873fa5-6e7b-4b32-81b7-35a06abc2d81\") " pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:16:34.415399 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.415374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdtc5\" (UniqueName: \"kubernetes.io/projected/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-kube-api-access-wdtc5\") pod \"payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d\" (UID: \"1d873fa5-6e7b-4b32-81b7-35a06abc2d81\") " pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:16:34.512429 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.512403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:16:34.790915 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.790890    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"]
Sep 14 03:16:34.791483 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:16:34.791460    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d873fa5_6e7b_4b32_81b7_35a06abc2d81.slice/crio-5d2df0da701f9def525944a5475234bbf368802a43e4aff6a35c969999214267 WatchSource:0}: Error finding container 5d2df0da701f9def525944a5475234bbf368802a43e4aff6a35c969999214267: Status 404 returned error can't find the container with id 5d2df0da701f9def525944a5475234bbf368802a43e4aff6a35c969999214267
Sep 14 03:16:34.926850 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.926818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d" event={"ID":"1d873fa5-6e7b-4b32-81b7-35a06abc2d81","Type":"ContainerStarted","Data":"d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78"}
Sep 14 03:16:34.927021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.926889    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d" event={"ID":"1d873fa5-6e7b-4b32-81b7-35a06abc2d81","Type":"ContainerStarted","Data":"5d2df0da701f9def525944a5475234bbf368802a43e4aff6a35c969999214267"}
Sep 14 03:16:34.927021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.926943    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:16:34.945490 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:34.945448    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d" podStartSLOduration=0.945435159 podStartE2EDuration="945.435159ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:16:34.942734838 +0000 UTC m=+1461.668692890" watchObservedRunningTime="2026-09-14 03:16:34.945435159 +0000 UTC m=+1461.671393257"
Sep 14 03:16:37.662622 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:37.662590    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:16:45.936109 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:45.936076    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:16:46.349970 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.349931    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5" podUID="d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" containerName="main" containerID="cri-o://ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e" gracePeriod=45
Sep 14 03:16:46.356915 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.356858    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg" podUID="d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" containerName="main" containerID="cri-o://0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65" gracePeriod=45
Sep 14 03:16:46.379169 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.379139    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5" podUID="07863119-ad92-4bf2-bb9e-8b851262fbe3" containerName="main" containerID="cri-o://a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382" gracePeriod=45
Sep 14 03:16:46.383458 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.383430    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn" podUID="32150566-1bac-4eca-b69f-1b997aeefbbe" containerName="main" containerID="cri-o://c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a" gracePeriod=45
Sep 14 03:16:46.633579 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.633551    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:16:46.678143 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.678116    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:16:46.723078 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.723013    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:16:46.726346 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726322    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-tmp-dir\") pod \"07863119-ad92-4bf2-bb9e-8b851262fbe3\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") "
Sep 14 03:16:46.726446 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726365    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-home\") pod \"07863119-ad92-4bf2-bb9e-8b851262fbe3\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") "
Sep 14 03:16:46.726446 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726390    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-dshm\") pod \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") "
Sep 14 03:16:46.726446 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726419    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqsmv\" (UniqueName: \"kubernetes.io/projected/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-kube-api-access-qqsmv\") pod \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") "
Sep 14 03:16:46.726624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tmp-dir\") pod \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") "
Sep 14 03:16:46.726624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tls-certs\") pod \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") "
Sep 14 03:16:46.726624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07863119-ad92-4bf2-bb9e-8b851262fbe3-tls-certs\") pod \"07863119-ad92-4bf2-bb9e-8b851262fbe3\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") "
Sep 14 03:16:46.726624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726535    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc4fd\" (UniqueName: \"kubernetes.io/projected/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-kube-api-access-wc4fd\") pod \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") "
Sep 14 03:16:46.726624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tmp-dir\") pod \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") "
Sep 14 03:16:46.726624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726594    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07863119-ad92-4bf2-bb9e-8b851262fbe3" (UID: "07863119-ad92-4bf2-bb9e-8b851262fbe3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.726964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-model-cache\") pod \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") "
Sep 14 03:16:46.726964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-dshm\") pod \"07863119-ad92-4bf2-bb9e-8b851262fbe3\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") "
Sep 14 03:16:46.726964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-home\") pod \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") "
Sep 14 03:16:46.726964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-home\") pod \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") "
Sep 14 03:16:46.726964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726757    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" (UID: "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.726964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726818    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnmp4\" (UniqueName: \"kubernetes.io/projected/07863119-ad92-4bf2-bb9e-8b851262fbe3-kube-api-access-cnmp4\") pod \"07863119-ad92-4bf2-bb9e-8b851262fbe3\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") "
Sep 14 03:16:46.726964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726857    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-model-cache\") pod \"07863119-ad92-4bf2-bb9e-8b851262fbe3\" (UID: \"07863119-ad92-4bf2-bb9e-8b851262fbe3\") "
Sep 14 03:16:46.726964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726933    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-home" (OuterVolumeSpecName: "home") pod "07863119-ad92-4bf2-bb9e-8b851262fbe3" (UID: "07863119-ad92-4bf2-bb9e-8b851262fbe3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.727416 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.726997    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-model-cache\") pod \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\" (UID: \"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b\") "
Sep 14 03:16:46.727416 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tls-certs\") pod \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") "
Sep 14 03:16:46.727416 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-dshm\") pod \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\" (UID: \"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf\") "
Sep 14 03:16:46.727416 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727155    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" (UID: "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.727637 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727419    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-home" (OuterVolumeSpecName: "home") pod "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" (UID: "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.727637 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727502    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" (UID: "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.727637 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727546    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.727637 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727563    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.727637 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727576    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.727637 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727588    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.727637 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727603    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.728018 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727705    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-home" (OuterVolumeSpecName: "home") pod "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" (UID: "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.728018 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.727717    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-model-cache" (OuterVolumeSpecName: "model-cache") pod "07863119-ad92-4bf2-bb9e-8b851262fbe3" (UID: "07863119-ad92-4bf2-bb9e-8b851262fbe3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.728264 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.728153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" (UID: "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.729630 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.729603    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-dshm" (OuterVolumeSpecName: "dshm") pod "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" (UID: "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.729973 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.729932    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07863119-ad92-4bf2-bb9e-8b851262fbe3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07863119-ad92-4bf2-bb9e-8b851262fbe3" (UID: "07863119-ad92-4bf2-bb9e-8b851262fbe3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:16:46.730569 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.730538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07863119-ad92-4bf2-bb9e-8b851262fbe3-kube-api-access-cnmp4" (OuterVolumeSpecName: "kube-api-access-cnmp4") pod "07863119-ad92-4bf2-bb9e-8b851262fbe3" (UID: "07863119-ad92-4bf2-bb9e-8b851262fbe3"). InnerVolumeSpecName "kube-api-access-cnmp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:16:46.730569 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.730551    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:16:46.730751 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.730564    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-dshm" (OuterVolumeSpecName: "dshm") pod "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" (UID: "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.730751 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.730615    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-kube-api-access-qqsmv" (OuterVolumeSpecName: "kube-api-access-qqsmv") pod "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" (UID: "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf"). InnerVolumeSpecName "kube-api-access-qqsmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:16:46.730919 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.730902    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" (UID: "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:16:46.731165 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.731146    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-dshm" (OuterVolumeSpecName: "dshm") pod "07863119-ad92-4bf2-bb9e-8b851262fbe3" (UID: "07863119-ad92-4bf2-bb9e-8b851262fbe3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.731356 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.731332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-kube-api-access-wc4fd" (OuterVolumeSpecName: "kube-api-access-wc4fd") pod "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" (UID: "d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b"). InnerVolumeSpecName "kube-api-access-wc4fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:16:46.731499 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.731481    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" (UID: "d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:16:46.828150 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828119    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-model-cache\") pod \"32150566-1bac-4eca-b69f-1b997aeefbbe\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") "
Sep 14 03:16:46.828342 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828163    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgtf5\" (UniqueName: \"kubernetes.io/projected/32150566-1bac-4eca-b69f-1b997aeefbbe-kube-api-access-bgtf5\") pod \"32150566-1bac-4eca-b69f-1b997aeefbbe\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") "
Sep 14 03:16:46.828342 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828229    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-home\") pod \"32150566-1bac-4eca-b69f-1b997aeefbbe\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") "
Sep 14 03:16:46.828342 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828257    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32150566-1bac-4eca-b69f-1b997aeefbbe-tls-certs\") pod \"32150566-1bac-4eca-b69f-1b997aeefbbe\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") "
Sep 14 03:16:46.828342 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828285    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-dshm\") pod \"32150566-1bac-4eca-b69f-1b997aeefbbe\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") "
Sep 14 03:16:46.828342 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-tmp-dir\") pod \"32150566-1bac-4eca-b69f-1b997aeefbbe\" (UID: \"32150566-1bac-4eca-b69f-1b997aeefbbe\") "
Sep 14 03:16:46.828621 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828369    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "32150566-1bac-4eca-b69f-1b997aeefbbe" (UID: "32150566-1bac-4eca-b69f-1b997aeefbbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.828621 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828512    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-home" (OuterVolumeSpecName: "home") pod "32150566-1bac-4eca-b69f-1b997aeefbbe" (UID: "32150566-1bac-4eca-b69f-1b997aeefbbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.828621 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828607    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32150566-1bac-4eca-b69f-1b997aeefbbe" (UID: "32150566-1bac-4eca-b69f-1b997aeefbbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.828778 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828761    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828822 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828785    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828822 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828801    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnmp4\" (UniqueName: \"kubernetes.io/projected/07863119-ad92-4bf2-bb9e-8b851262fbe3-kube-api-access-cnmp4\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828822 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828815    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828829    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07863119-ad92-4bf2-bb9e-8b851262fbe3-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828842    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828855    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828867    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828878    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828889    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828903    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqsmv\" (UniqueName: \"kubernetes.io/projected/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf-kube-api-access-qqsmv\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828914    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828927    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07863119-ad92-4bf2-bb9e-8b851262fbe3-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.828943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828939    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc4fd\" (UniqueName: \"kubernetes.io/projected/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-kube-api-access-wc4fd\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.829264 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828951    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.829264 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.828964    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.830325 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.830305    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-dshm" (OuterVolumeSpecName: "dshm") pod "32150566-1bac-4eca-b69f-1b997aeefbbe" (UID: "32150566-1bac-4eca-b69f-1b997aeefbbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.830406 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.830352    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32150566-1bac-4eca-b69f-1b997aeefbbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32150566-1bac-4eca-b69f-1b997aeefbbe" (UID: "32150566-1bac-4eca-b69f-1b997aeefbbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:16:46.830406 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.830383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32150566-1bac-4eca-b69f-1b997aeefbbe-kube-api-access-bgtf5" (OuterVolumeSpecName: "kube-api-access-bgtf5") pod "32150566-1bac-4eca-b69f-1b997aeefbbe" (UID: "32150566-1bac-4eca-b69f-1b997aeefbbe"). InnerVolumeSpecName "kube-api-access-bgtf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:16:46.930483 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.930412    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32150566-1bac-4eca-b69f-1b997aeefbbe-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.930483 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.930436    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32150566-1bac-4eca-b69f-1b997aeefbbe-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.930483 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:46.930447    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgtf5\" (UniqueName: \"kubernetes.io/projected/32150566-1bac-4eca-b69f-1b997aeefbbe-kube-api-access-bgtf5\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:16:47.005777 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.005666    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" containerID="ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e" exitCode=0
Sep 14 03:16:47.005777 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.005772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5" event={"ID":"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b","Type":"ContainerDied","Data":"ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e"}
Sep 14 03:16:47.006473 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.005810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5" event={"ID":"d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b","Type":"ContainerDied","Data":"a96449725c76177b3588d7e95fe89d1e8137c1b9bcfdf044f64cc8d7d3f11a26"}
Sep 14 03:16:47.006473 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.005831    2580 scope.go:117] "RemoveContainer" containerID="ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e"
Sep 14 03:16:47.006473 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.005984    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"
Sep 14 03:16:47.009508 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.009380    2580 generic.go:356] "Generic (PLEG): container finished" podID="d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" containerID="0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65" exitCode=0
Sep 14 03:16:47.009508 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.009471    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"
Sep 14 03:16:47.009508 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.009495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg" event={"ID":"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf","Type":"ContainerDied","Data":"0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65"}
Sep 14 03:16:47.009740 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.009568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg" event={"ID":"d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf","Type":"ContainerDied","Data":"377d500f1b2fe8d4924471e56ae474b415207a49d8bf649dffc371add913762c"}
Sep 14 03:16:47.012249 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.012221    2580 generic.go:356] "Generic (PLEG): container finished" podID="32150566-1bac-4eca-b69f-1b997aeefbbe" containerID="c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a" exitCode=0
Sep 14 03:16:47.012355 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.012305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn" event={"ID":"32150566-1bac-4eca-b69f-1b997aeefbbe","Type":"ContainerDied","Data":"c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a"}
Sep 14 03:16:47.012355 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.012335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn" event={"ID":"32150566-1bac-4eca-b69f-1b997aeefbbe","Type":"ContainerDied","Data":"fb2a47a91c754d1bb61503c995e94bda5ecbae12c658f983d3de449c9522abd4"}
Sep 14 03:16:47.012485 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.012395    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"
Sep 14 03:16:47.014119 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.014083    2580 generic.go:356] "Generic (PLEG): container finished" podID="07863119-ad92-4bf2-bb9e-8b851262fbe3" containerID="a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382" exitCode=0
Sep 14 03:16:47.014119 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.014117    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"
Sep 14 03:16:47.014296 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.014164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5" event={"ID":"07863119-ad92-4bf2-bb9e-8b851262fbe3","Type":"ContainerDied","Data":"a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382"}
Sep 14 03:16:47.014296 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.014211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5" event={"ID":"07863119-ad92-4bf2-bb9e-8b851262fbe3","Type":"ContainerDied","Data":"36be7669675c4ac9ca3c28cfc963ea94d266889b439a0740cbd49c09a327de66"}
Sep 14 03:16:47.019400 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.019380    2580 scope.go:117] "RemoveContainer" containerID="ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e"
Sep 14 03:16:47.019707 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:16:47.019678    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e\": container with ID starting with ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e not found: ID does not exist" containerID="ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e"
Sep 14 03:16:47.019810 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.019718    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e"} err="failed to get container status \"ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e\": rpc error: code = NotFound desc = could not find container \"ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e\": container with ID starting with ef6101c21b8d4738369ce71a37fa066efe9c33b512cbc71dc64115a3edd1fd7e not found: ID does not exist"
Sep 14 03:16:47.019810 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.019742    2580 scope.go:117] "RemoveContainer" containerID="0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65"
Sep 14 03:16:47.029315 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.029299    2580 scope.go:117] "RemoveContainer" containerID="0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65"
Sep 14 03:16:47.029571 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:16:47.029555    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65\": container with ID starting with 0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65 not found: ID does not exist" containerID="0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65"
Sep 14 03:16:47.029622 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.029577    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65"} err="failed to get container status \"0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65\": rpc error: code = NotFound desc = could not find container \"0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65\": container with ID starting with 0b4e59c011a9101938ae60ba1bc23b9b87959f0822b1f2219f995534782c8d65 not found: ID does not exist"
Sep 14 03:16:47.029622 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.029592    2580 scope.go:117] "RemoveContainer" containerID="c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a"
Sep 14 03:16:47.037714 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.037696    2580 scope.go:117] "RemoveContainer" containerID="c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a"
Sep 14 03:16:47.037926 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:16:47.037903    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a\": container with ID starting with c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a not found: ID does not exist" containerID="c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a"
Sep 14 03:16:47.037976 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.037930    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a"} err="failed to get container status \"c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a\": rpc error: code = NotFound desc = could not find container \"c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a\": container with ID starting with c2a2bb7adc2c4b342611df502b68724bf4ef6707782ac2ef471284097836d61a not found: ID does not exist"
Sep 14 03:16:47.037976 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.037943    2580 scope.go:117] "RemoveContainer" containerID="a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382"
Sep 14 03:16:47.045708 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.045683    2580 scope.go:117] "RemoveContainer" containerID="a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382"
Sep 14 03:16:47.045945 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:16:47.045928    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382\": container with ID starting with a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382 not found: ID does not exist" containerID="a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382"
Sep 14 03:16:47.046007 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.045952    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382"} err="failed to get container status \"a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382\": rpc error: code = NotFound desc = could not find container \"a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382\": container with ID starting with a73a6a8d554b151b3d199f89d0917a237d3f6429481c69148f62cc4669662382 not found: ID does not exist"
Sep 14 03:16:47.067874 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.067851    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"]
Sep 14 03:16:47.073989 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.073967    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-2-w0-3a752f-kserve-867cf7455b-5zxm5"]
Sep 14 03:16:47.088125 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.088106    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"]
Sep 14 03:16:47.095066 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.095046    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/simulator-w0-3a752f-kserve-7c8f95b86f-pvtrn"]
Sep 14 03:16:47.108656 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.108633    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"]
Sep 14 03:16:47.116695 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.116676    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/premium-w0-3a752f-kserve-769b69d8c6-htrs5"]
Sep 14 03:16:47.136383 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.136358    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"]
Sep 14 03:16:47.142275 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.142253    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/distinct-w0-3a752f-kserve-7896cc46c9-kk7bg"]
Sep 14 03:16:47.757158 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.757131    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07863119-ad92-4bf2-bb9e-8b851262fbe3" path="/var/lib/kubelet/pods/07863119-ad92-4bf2-bb9e-8b851262fbe3/volumes"
Sep 14 03:16:47.757522 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.757509    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32150566-1bac-4eca-b69f-1b997aeefbbe" path="/var/lib/kubelet/pods/32150566-1bac-4eca-b69f-1b997aeefbbe/volumes"
Sep 14 03:16:47.757856 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.757844    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" path="/var/lib/kubelet/pods/d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf/volumes"
Sep 14 03:16:47.758504 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:16:47.758489    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" path="/var/lib/kubelet/pods/d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b/volumes"
Sep 14 03:17:10.164311 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:10.164238    2580 generic.go:356] "Generic (PLEG): container finished" podID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerID="c5f72ef5f3d03cd88e29f85b34e1c3ee8aa759e72333a71d3ff3a50001ba0d1f" exitCode=0
Sep 14 03:17:10.164785 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:10.164315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" event={"ID":"b5abadfa-a761-43fe-afe6-9ab492992fb9","Type":"ContainerDied","Data":"c5f72ef5f3d03cd88e29f85b34e1c3ee8aa759e72333a71d3ff3a50001ba0d1f"}
Sep 14 03:17:10.164853 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:10.164828    2580 scope.go:117] "RemoveContainer" containerID="c5f72ef5f3d03cd88e29f85b34e1c3ee8aa759e72333a71d3ff3a50001ba0d1f"
Sep 14 03:17:10.166019 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:10.165990    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:17:10.166101 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:10.166083    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:17:10.166139 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:10.166117    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:17:10.866838 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:10.866809    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:17:11.172023 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:11.171938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" event={"ID":"b5abadfa-a761-43fe-afe6-9ab492992fb9","Type":"ContainerStarted","Data":"652a5a8f3a11ce32548658428829ff07dd04ec963427c26bfbe1c1e53e2830c3"}
Sep 14 03:17:12.020011 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:12.019980    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:17:12.020247 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:12.020151    2580 patch_prober.go:28] interesting pod/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 14 03:17:12.020340 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:12.020283    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 14 03:17:13.019836 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:13.019799    2580 patch_prober.go:28] interesting pod/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 14 03:17:13.020406 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:13.020376    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 14 03:17:13.853195 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:13.853169    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:17:13.853195 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:13.853194    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:17:14.019172 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:14.019136    2580 patch_prober.go:28] interesting pod/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 14 03:17:14.019327 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:14.019200    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 14 03:17:15.019402 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:15.019363    2580 patch_prober.go:28] interesting pod/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 14 03:17:15.019786 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:15.019416    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 14 03:17:16.019466 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:16.019424    2580 patch_prober.go:28] interesting pod/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 14 03:17:16.019884 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:16.019487    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 14 03:17:17.022606 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:17.022577    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:17:17.023107 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:17.022875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:17:17.023571 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:17.023551    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:17:19.173259 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:19.173220    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:17:29.346985 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:29.346937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"]
Sep 14 03:17:29.348173 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:29.347200    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d" podUID="1d873fa5-6e7b-4b32-81b7-35a06abc2d81" containerName="payload-processing" containerID="cri-o://d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78" gracePeriod=30
Sep 14 03:17:29.588105 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:29.588081    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:17:29.669081 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:29.668997    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-plugins-config-volume\") pod \"1d873fa5-6e7b-4b32-81b7-35a06abc2d81\" (UID: \"1d873fa5-6e7b-4b32-81b7-35a06abc2d81\") "
Sep 14 03:17:29.669229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:29.669105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdtc5\" (UniqueName: \"kubernetes.io/projected/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-kube-api-access-wdtc5\") pod \"1d873fa5-6e7b-4b32-81b7-35a06abc2d81\" (UID: \"1d873fa5-6e7b-4b32-81b7-35a06abc2d81\") "
Sep 14 03:17:29.669351 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:29.669327    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d873fa5-6e7b-4b32-81b7-35a06abc2d81" (UID: "1d873fa5-6e7b-4b32-81b7-35a06abc2d81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:17:29.671174 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:29.671149    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-kube-api-access-wdtc5" (OuterVolumeSpecName: "kube-api-access-wdtc5") pod "1d873fa5-6e7b-4b32-81b7-35a06abc2d81" (UID: "1d873fa5-6e7b-4b32-81b7-35a06abc2d81"). InnerVolumeSpecName "kube-api-access-wdtc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:17:29.769773 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:29.769748    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:29.769773 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:29.769773    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdtc5\" (UniqueName: \"kubernetes.io/projected/1d873fa5-6e7b-4b32-81b7-35a06abc2d81-kube-api-access-wdtc5\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:30.281856 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:30.281819    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d873fa5-6e7b-4b32-81b7-35a06abc2d81" containerID="d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78" exitCode=0
Sep 14 03:17:30.282043 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:30.281860    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"
Sep 14 03:17:30.282043 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:30.281909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d" event={"ID":"1d873fa5-6e7b-4b32-81b7-35a06abc2d81","Type":"ContainerDied","Data":"d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78"}
Sep 14 03:17:30.282043 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:30.281964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d" event={"ID":"1d873fa5-6e7b-4b32-81b7-35a06abc2d81","Type":"ContainerDied","Data":"5d2df0da701f9def525944a5475234bbf368802a43e4aff6a35c969999214267"}
Sep 14 03:17:30.282043 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:30.281991    2580 scope.go:117] "RemoveContainer" containerID="d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78"
Sep 14 03:17:30.291222 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:30.291196    2580 scope.go:117] "RemoveContainer" containerID="d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78"
Sep 14 03:17:30.291469 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:17:30.291451    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78\": container with ID starting with d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78 not found: ID does not exist" containerID="d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78"
Sep 14 03:17:30.291541 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:30.291478    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78"} err="failed to get container status \"d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78\": rpc error: code = NotFound desc = could not find container \"d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78\": container with ID starting with d61685f6d272e37e47b8e7472a172b315a9f1606a0d91ab27fd8c91d4cd4ad78 not found: ID does not exist"
Sep 14 03:17:30.297725 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:30.297703    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"]
Sep 14 03:17:30.304606 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:30.304584    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97da78-85857b5bf-r5b5d"]
Sep 14 03:17:31.762200 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:31.760696    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d873fa5-6e7b-4b32-81b7-35a06abc2d81" path="/var/lib/kubelet/pods/1d873fa5-6e7b-4b32-81b7-35a06abc2d81/volumes"
Sep 14 03:17:36.183181 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:36.183096    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"]
Sep 14 03:17:36.183624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:36.183334    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy" containerID="cri-o://652a5a8f3a11ce32548658428829ff07dd04ec963427c26bfbe1c1e53e2830c3" gracePeriod=30
Sep 14 03:17:41.352196 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.351568    2580 generic.go:356] "Generic (PLEG): container finished" podID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerID="652a5a8f3a11ce32548658428829ff07dd04ec963427c26bfbe1c1e53e2830c3" exitCode=0
Sep 14 03:17:41.352196 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.351655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" event={"ID":"b5abadfa-a761-43fe-afe6-9ab492992fb9","Type":"ContainerDied","Data":"652a5a8f3a11ce32548658428829ff07dd04ec963427c26bfbe1c1e53e2830c3"}
Sep 14 03:17:41.352196 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.351712    2580 scope.go:117] "RemoveContainer" containerID="c5f72ef5f3d03cd88e29f85b34e1c3ee8aa759e72333a71d3ff3a50001ba0d1f"
Sep 14 03:17:41.425117 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.425093    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:17:41.579480 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.579448    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgxxp\" (UniqueName: \"kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-kube-api-access-kgxxp\") pod \"b5abadfa-a761-43fe-afe6-9ab492992fb9\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") "
Sep 14 03:17:41.579661 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.579503    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-podinfo\") pod \"b5abadfa-a761-43fe-afe6-9ab492992fb9\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") "
Sep 14 03:17:41.579661 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.579526    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-token\") pod \"b5abadfa-a761-43fe-afe6-9ab492992fb9\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") "
Sep 14 03:17:41.579661 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.579626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-certs\") pod \"b5abadfa-a761-43fe-afe6-9ab492992fb9\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") "
Sep 14 03:17:41.579661 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.579661    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-envoy\") pod \"b5abadfa-a761-43fe-afe6-9ab492992fb9\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") "
Sep 14 03:17:41.579923 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.579687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-data\") pod \"b5abadfa-a761-43fe-afe6-9ab492992fb9\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") "
Sep 14 03:17:41.579923 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.579742    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5abadfa-a761-43fe-afe6-9ab492992fb9-istiod-ca-cert\") pod \"b5abadfa-a761-43fe-afe6-9ab492992fb9\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") "
Sep 14 03:17:41.579923 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.579849    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-credential-socket\") pod \"b5abadfa-a761-43fe-afe6-9ab492992fb9\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") "
Sep 14 03:17:41.579923 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.579901    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-socket\") pod \"b5abadfa-a761-43fe-afe6-9ab492992fb9\" (UID: \"b5abadfa-a761-43fe-afe6-9ab492992fb9\") "
Sep 14 03:17:41.580168 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.579923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b5abadfa-a761-43fe-afe6-9ab492992fb9" (UID: "b5abadfa-a761-43fe-afe6-9ab492992fb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:17:41.580168 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.579998    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-data" (OuterVolumeSpecName: "istio-data") pod "b5abadfa-a761-43fe-afe6-9ab492992fb9" (UID: "b5abadfa-a761-43fe-afe6-9ab492992fb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:17:41.580285 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.580245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b5abadfa-a761-43fe-afe6-9ab492992fb9" (UID: "b5abadfa-a761-43fe-afe6-9ab492992fb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:17:41.580285 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.580262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5abadfa-a761-43fe-afe6-9ab492992fb9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b5abadfa-a761-43fe-afe6-9ab492992fb9" (UID: "b5abadfa-a761-43fe-afe6-9ab492992fb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:17:41.580395 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.580358    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:41.580395 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.580380    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:41.580509 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.580452    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b5abadfa-a761-43fe-afe6-9ab492992fb9" (UID: "b5abadfa-a761-43fe-afe6-9ab492992fb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:17:41.580509 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.580496    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5abadfa-a761-43fe-afe6-9ab492992fb9-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:41.580631 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.580512    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:41.581879 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.581855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b5abadfa-a761-43fe-afe6-9ab492992fb9" (UID: "b5abadfa-a761-43fe-afe6-9ab492992fb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 03:17:41.581879 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.581861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b5abadfa-a761-43fe-afe6-9ab492992fb9" (UID: "b5abadfa-a761-43fe-afe6-9ab492992fb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:17:41.582022 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.581879    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-kube-api-access-kgxxp" (OuterVolumeSpecName: "kube-api-access-kgxxp") pod "b5abadfa-a761-43fe-afe6-9ab492992fb9" (UID: "b5abadfa-a761-43fe-afe6-9ab492992fb9"). InnerVolumeSpecName "kube-api-access-kgxxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:17:41.582022 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.581902    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-token" (OuterVolumeSpecName: "istio-token") pod "b5abadfa-a761-43fe-afe6-9ab492992fb9" (UID: "b5abadfa-a761-43fe-afe6-9ab492992fb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:17:41.681656 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.681627    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:41.681656 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.681655    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgxxp\" (UniqueName: \"kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-kube-api-access-kgxxp\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:41.681852 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.681671    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:41.681852 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.681683    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:41.681852 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:41.681694    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5abadfa-a761-43fe-afe6-9ab492992fb9-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:42.358432 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:42.358401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh" event={"ID":"b5abadfa-a761-43fe-afe6-9ab492992fb9","Type":"ContainerDied","Data":"9f52c02bc8adb7c28715a19b732ced8a94155f36483caceb7bde05127f12598d"}
Sep 14 03:17:42.358847 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:42.358453    2580 scope.go:117] "RemoveContainer" containerID="652a5a8f3a11ce32548658428829ff07dd04ec963427c26bfbe1c1e53e2830c3"
Sep 14 03:17:42.358847 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:42.358415    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"
Sep 14 03:17:42.378348 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:42.378324    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"]
Sep 14 03:17:42.386465 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:42.386442    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-97da78-openshift-default-6bb96878c4-wgvrh"]
Sep 14 03:17:43.756873 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:43.756840    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" path="/var/lib/kubelet/pods/b5abadfa-a761-43fe-afe6-9ab492992fb9/volumes"
Sep 14 03:17:46.990930 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:46.990896    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"]
Sep 14 03:17:46.991402 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:46.991175    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82" podUID="9c07fba8-b58c-424d-8861-7da5cce15c8e" containerName="maas-api" containerID="cri-o://d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64" gracePeriod=30
Sep 14 03:17:47.232977 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.232954    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:17:47.332320 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.332292    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c07fba8-b58c-424d-8861-7da5cce15c8e-openshift-service-ca\") pod \"9c07fba8-b58c-424d-8861-7da5cce15c8e\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") "
Sep 14 03:17:47.332582 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.332327    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-metrics-certs\") pod \"9c07fba8-b58c-424d-8861-7da5cce15c8e\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") "
Sep 14 03:17:47.332582 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.332356    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph74v\" (UniqueName: \"kubernetes.io/projected/9c07fba8-b58c-424d-8861-7da5cce15c8e-kube-api-access-ph74v\") pod \"9c07fba8-b58c-424d-8861-7da5cce15c8e\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") "
Sep 14 03:17:47.332582 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.332421    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-maas-api-tls\") pod \"9c07fba8-b58c-424d-8861-7da5cce15c8e\" (UID: \"9c07fba8-b58c-424d-8861-7da5cce15c8e\") "
Sep 14 03:17:47.332787 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.332635    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c07fba8-b58c-424d-8861-7da5cce15c8e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c07fba8-b58c-424d-8861-7da5cce15c8e" (UID: "9c07fba8-b58c-424d-8861-7da5cce15c8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:17:47.334475 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.334443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c07fba8-b58c-424d-8861-7da5cce15c8e" (UID: "9c07fba8-b58c-424d-8861-7da5cce15c8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:17:47.334475 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.334451    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c07fba8-b58c-424d-8861-7da5cce15c8e" (UID: "9c07fba8-b58c-424d-8861-7da5cce15c8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:17:47.334642 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.334492    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c07fba8-b58c-424d-8861-7da5cce15c8e-kube-api-access-ph74v" (OuterVolumeSpecName: "kube-api-access-ph74v") pod "9c07fba8-b58c-424d-8861-7da5cce15c8e" (UID: "9c07fba8-b58c-424d-8861-7da5cce15c8e"). InnerVolumeSpecName "kube-api-access-ph74v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:17:47.388913 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.388883    2580 generic.go:356] "Generic (PLEG): container finished" podID="9c07fba8-b58c-424d-8861-7da5cce15c8e" containerID="d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64" exitCode=0
Sep 14 03:17:47.389009 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.388929    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"
Sep 14 03:17:47.389009 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.388968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82" event={"ID":"9c07fba8-b58c-424d-8861-7da5cce15c8e","Type":"ContainerDied","Data":"d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64"}
Sep 14 03:17:47.389115 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.389022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82" event={"ID":"9c07fba8-b58c-424d-8861-7da5cce15c8e","Type":"ContainerDied","Data":"df1af8dc90f783b3f21c32d11f8844baae19585f46f192894d604105818cbd36"}
Sep 14 03:17:47.389115 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.389059    2580 scope.go:117] "RemoveContainer" containerID="d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64"
Sep 14 03:17:47.399087 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.399072    2580 scope.go:117] "RemoveContainer" containerID="d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64"
Sep 14 03:17:47.399344 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:17:47.399324    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64\": container with ID starting with d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64 not found: ID does not exist" containerID="d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64"
Sep 14 03:17:47.399401 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.399353    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64"} err="failed to get container status \"d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64\": rpc error: code = NotFound desc = could not find container \"d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64\": container with ID starting with d8b326e0c017f65d1b840a056c33761b4da33274e0276d3d2feedea6f02a6b64 not found: ID does not exist"
Sep 14 03:17:47.413862 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.413793    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"]
Sep 14 03:17:47.416670 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.416651    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ce5b9b-6d54f4df77-zlx82"]
Sep 14 03:17:47.433481 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.433460    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c07fba8-b58c-424d-8861-7da5cce15c8e-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:47.433602 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.433485    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:47.433602 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.433502    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph74v\" (UniqueName: \"kubernetes.io/projected/9c07fba8-b58c-424d-8861-7da5cce15c8e-kube-api-access-ph74v\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:47.433602 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.433516    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c07fba8-b58c-424d-8861-7da5cce15c8e-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:17:47.758139 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:47.758107    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c07fba8-b58c-424d-8861-7da5cce15c8e" path="/var/lib/kubelet/pods/9c07fba8-b58c-424d-8861-7da5cce15c8e/volumes"
Sep 14 03:17:49.363737 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:49.363705    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"]
Sep 14 03:17:53.259823 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:17:53.259784    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:18:04.382821 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.382758    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m" podUID="0c500aa2-1218-430f-9e91-0325ec0a08de" containerName="main" containerID="cri-o://d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9" gracePeriod=45
Sep 14 03:18:04.632433 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.632407    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:18:04.686872 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.686798    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-tmp-dir\") pod \"0c500aa2-1218-430f-9e91-0325ec0a08de\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") "
Sep 14 03:18:04.686872 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.686851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-model-cache\") pod \"0c500aa2-1218-430f-9e91-0325ec0a08de\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") "
Sep 14 03:18:04.687138 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.686935    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-home\") pod \"0c500aa2-1218-430f-9e91-0325ec0a08de\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") "
Sep 14 03:18:04.687138 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.686967    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c500aa2-1218-430f-9e91-0325ec0a08de-tls-certs\") pod \"0c500aa2-1218-430f-9e91-0325ec0a08de\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") "
Sep 14 03:18:04.687138 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.687023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-dshm\") pod \"0c500aa2-1218-430f-9e91-0325ec0a08de\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") "
Sep 14 03:18:04.687138 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.687106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4v6q\" (UniqueName: \"kubernetes.io/projected/0c500aa2-1218-430f-9e91-0325ec0a08de-kube-api-access-k4v6q\") pod \"0c500aa2-1218-430f-9e91-0325ec0a08de\" (UID: \"0c500aa2-1218-430f-9e91-0325ec0a08de\") "
Sep 14 03:18:04.687138 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.687124    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c500aa2-1218-430f-9e91-0325ec0a08de" (UID: "0c500aa2-1218-430f-9e91-0325ec0a08de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.687410 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.687183    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c500aa2-1218-430f-9e91-0325ec0a08de" (UID: "0c500aa2-1218-430f-9e91-0325ec0a08de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.687410 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.687263    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-home" (OuterVolumeSpecName: "home") pod "0c500aa2-1218-430f-9e91-0325ec0a08de" (UID: "0c500aa2-1218-430f-9e91-0325ec0a08de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.687527 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.687431    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.687527 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.687473    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.687527 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.687487    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.689249 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.689221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-dshm" (OuterVolumeSpecName: "dshm") pod "0c500aa2-1218-430f-9e91-0325ec0a08de" (UID: "0c500aa2-1218-430f-9e91-0325ec0a08de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.689354 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.689262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c500aa2-1218-430f-9e91-0325ec0a08de-kube-api-access-k4v6q" (OuterVolumeSpecName: "kube-api-access-k4v6q") pod "0c500aa2-1218-430f-9e91-0325ec0a08de" (UID: "0c500aa2-1218-430f-9e91-0325ec0a08de"). InnerVolumeSpecName "kube-api-access-k4v6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:18:04.689441 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.689427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c500aa2-1218-430f-9e91-0325ec0a08de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c500aa2-1218-430f-9e91-0325ec0a08de" (UID: "0c500aa2-1218-430f-9e91-0325ec0a08de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:18:04.788617 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.788585    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4v6q\" (UniqueName: \"kubernetes.io/projected/0c500aa2-1218-430f-9e91-0325ec0a08de-kube-api-access-k4v6q\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.788617 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.788618    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c500aa2-1218-430f-9e91-0325ec0a08de-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.788809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:04.788634    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c500aa2-1218-430f-9e91-0325ec0a08de-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:05.494640 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:05.494601    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c500aa2-1218-430f-9e91-0325ec0a08de" containerID="d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9" exitCode=0
Sep 14 03:18:05.495049 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:05.494652    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"
Sep 14 03:18:05.495049 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:05.494655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m" event={"ID":"0c500aa2-1218-430f-9e91-0325ec0a08de","Type":"ContainerDied","Data":"d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9"}
Sep 14 03:18:05.495049 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:05.494770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m" event={"ID":"0c500aa2-1218-430f-9e91-0325ec0a08de","Type":"ContainerDied","Data":"03f6d07b920b76a01b3a03f565271e4a0988fada9f2fe90dfe39768ec6bc68e8"}
Sep 14 03:18:05.495049 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:05.494791    2580 scope.go:117] "RemoveContainer" containerID="d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9"
Sep 14 03:18:05.504690 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:05.504661    2580 scope.go:117] "RemoveContainer" containerID="d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9"
Sep 14 03:18:05.505304 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:18:05.504961    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9\": container with ID starting with d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9 not found: ID does not exist" containerID="d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9"
Sep 14 03:18:05.505304 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:05.504996    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9"} err="failed to get container status \"d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9\": rpc error: code = NotFound desc = could not find container \"d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9\": container with ID starting with d458f9115af6d380b6b53a1bf1deed2cad965f218a251cfb07ae0f6347cf1bb9 not found: ID does not exist"
Sep 14 03:18:05.519379 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:05.519357    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"]
Sep 14 03:18:05.523182 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:05.523162    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/simulator-w4-9c9f1e-kserve-86685774cd-clx2m"]
Sep 14 03:18:05.757280 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:05.757253    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c500aa2-1218-430f-9e91-0325ec0a08de" path="/var/lib/kubelet/pods/0c500aa2-1218-430f-9e91-0325ec0a08de/volumes"
Sep 14 03:18:07.471795 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.471758    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"]
Sep 14 03:18:07.472278 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.472023    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96" podUID="17441f5d-c1ae-41ed-acc1-1e3075d91234" containerName="maas-api" containerID="cri-o://fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6" gracePeriod=30
Sep 14 03:18:07.503043 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.503000    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq"]
Sep 14 03:18:07.507516 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.507493    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-17d8b8-29822595-dsrkq"]
Sep 14 03:18:07.530270 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.530245    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"]
Sep 14 03:18:07.530481 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.530437    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x" podUID="65bf6740-dca5-4443-8ac1-79500cb5427c" containerName="payload-processing" containerID="cri-o://007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0" gracePeriod=30
Sep 14 03:18:07.764289 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.764212    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33339fca-0aef-40d4-88de-7c717c527dd1" path="/var/lib/kubelet/pods/33339fca-0aef-40d4-88de-7c717c527dd1/volumes"
Sep 14 03:18:07.794900 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.794876    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:18:07.812476 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.812454    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17441f5d-c1ae-41ed-acc1-1e3075d91234-openshift-service-ca\") pod \"17441f5d-c1ae-41ed-acc1-1e3075d91234\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") "
Sep 14 03:18:07.812587 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.812514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-metrics-certs\") pod \"17441f5d-c1ae-41ed-acc1-1e3075d91234\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") "
Sep 14 03:18:07.812632 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.812607    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-maas-api-tls\") pod \"17441f5d-c1ae-41ed-acc1-1e3075d91234\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") "
Sep 14 03:18:07.812705 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.812687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6dvp\" (UniqueName: \"kubernetes.io/projected/17441f5d-c1ae-41ed-acc1-1e3075d91234-kube-api-access-z6dvp\") pod \"17441f5d-c1ae-41ed-acc1-1e3075d91234\" (UID: \"17441f5d-c1ae-41ed-acc1-1e3075d91234\") "
Sep 14 03:18:07.812794 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.812772    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17441f5d-c1ae-41ed-acc1-1e3075d91234-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17441f5d-c1ae-41ed-acc1-1e3075d91234" (UID: "17441f5d-c1ae-41ed-acc1-1e3075d91234"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:18:07.813121 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.813090    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17441f5d-c1ae-41ed-acc1-1e3075d91234-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:07.814797 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.814776    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17441f5d-c1ae-41ed-acc1-1e3075d91234" (UID: "17441f5d-c1ae-41ed-acc1-1e3075d91234"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:18:07.815024 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.815003    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "17441f5d-c1ae-41ed-acc1-1e3075d91234" (UID: "17441f5d-c1ae-41ed-acc1-1e3075d91234"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:18:07.815109 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.815018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17441f5d-c1ae-41ed-acc1-1e3075d91234-kube-api-access-z6dvp" (OuterVolumeSpecName: "kube-api-access-z6dvp") pod "17441f5d-c1ae-41ed-acc1-1e3075d91234" (UID: "17441f5d-c1ae-41ed-acc1-1e3075d91234"). InnerVolumeSpecName "kube-api-access-z6dvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:18:07.816387 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.816371    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:18:07.913381 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.913354    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkd56\" (UniqueName: \"kubernetes.io/projected/65bf6740-dca5-4443-8ac1-79500cb5427c-kube-api-access-wkd56\") pod \"65bf6740-dca5-4443-8ac1-79500cb5427c\" (UID: \"65bf6740-dca5-4443-8ac1-79500cb5427c\") "
Sep 14 03:18:07.913552 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.913454    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bf6740-dca5-4443-8ac1-79500cb5427c-plugins-config-volume\") pod \"65bf6740-dca5-4443-8ac1-79500cb5427c\" (UID: \"65bf6740-dca5-4443-8ac1-79500cb5427c\") "
Sep 14 03:18:07.913713 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.913699    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:07.913772 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.913714    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6dvp\" (UniqueName: \"kubernetes.io/projected/17441f5d-c1ae-41ed-acc1-1e3075d91234-kube-api-access-z6dvp\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:07.913772 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.913723    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17441f5d-c1ae-41ed-acc1-1e3075d91234-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:07.913899 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.913872    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bf6740-dca5-4443-8ac1-79500cb5427c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65bf6740-dca5-4443-8ac1-79500cb5427c" (UID: "65bf6740-dca5-4443-8ac1-79500cb5427c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:18:07.915438 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:07.915419    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bf6740-dca5-4443-8ac1-79500cb5427c-kube-api-access-wkd56" (OuterVolumeSpecName: "kube-api-access-wkd56") pod "65bf6740-dca5-4443-8ac1-79500cb5427c" (UID: "65bf6740-dca5-4443-8ac1-79500cb5427c"). InnerVolumeSpecName "kube-api-access-wkd56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:18:08.015233 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.015195    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bf6740-dca5-4443-8ac1-79500cb5427c-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:08.015233 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.015225    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkd56\" (UniqueName: \"kubernetes.io/projected/65bf6740-dca5-4443-8ac1-79500cb5427c-kube-api-access-wkd56\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:08.515770 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.515738    2580 generic.go:356] "Generic (PLEG): container finished" podID="65bf6740-dca5-4443-8ac1-79500cb5427c" containerID="007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0" exitCode=0
Sep 14 03:18:08.516231 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.515787    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"
Sep 14 03:18:08.516231 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.515832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x" event={"ID":"65bf6740-dca5-4443-8ac1-79500cb5427c","Type":"ContainerDied","Data":"007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0"}
Sep 14 03:18:08.516231 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.515904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x" event={"ID":"65bf6740-dca5-4443-8ac1-79500cb5427c","Type":"ContainerDied","Data":"094bc49087ee068b39989ad00a20bf61963cb2f6973276924b0d1586946f6a06"}
Sep 14 03:18:08.516231 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.515955    2580 scope.go:117] "RemoveContainer" containerID="007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0"
Sep 14 03:18:08.517343 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.517316    2580 generic.go:356] "Generic (PLEG): container finished" podID="17441f5d-c1ae-41ed-acc1-1e3075d91234" containerID="fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6" exitCode=0
Sep 14 03:18:08.517455 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.517360    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"
Sep 14 03:18:08.517455 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.517389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96" event={"ID":"17441f5d-c1ae-41ed-acc1-1e3075d91234","Type":"ContainerDied","Data":"fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6"}
Sep 14 03:18:08.517455 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.517427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96" event={"ID":"17441f5d-c1ae-41ed-acc1-1e3075d91234","Type":"ContainerDied","Data":"dc9115ce0d76a0923fcc9b275d759266912910313c34c9de9c92e9c200fca4da"}
Sep 14 03:18:08.525416 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.525272    2580 scope.go:117] "RemoveContainer" containerID="007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0"
Sep 14 03:18:08.525667 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:18:08.525645    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0\": container with ID starting with 007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0 not found: ID does not exist" containerID="007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0"
Sep 14 03:18:08.525726 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.525674    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0"} err="failed to get container status \"007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0\": rpc error: code = NotFound desc = could not find container \"007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0\": container with ID starting with 007826043c0f3d2f029852929452d2012a8688dc8af588cd7fc15fd36739c3c0 not found: ID does not exist"
Sep 14 03:18:08.525726 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.525691    2580 scope.go:117] "RemoveContainer" containerID="fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6"
Sep 14 03:18:08.533717 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.533698    2580 scope.go:117] "RemoveContainer" containerID="fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6"
Sep 14 03:18:08.533971 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:18:08.533953    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6\": container with ID starting with fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6 not found: ID does not exist" containerID="fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6"
Sep 14 03:18:08.534055 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.533976    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6"} err="failed to get container status \"fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6\": rpc error: code = NotFound desc = could not find container \"fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6\": container with ID starting with fb0971d89fc6381f859517dff177b3e85ea6e3539250504574c50feb7b5822e6 not found: ID does not exist"
Sep 14 03:18:08.542646 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.542625    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"]
Sep 14 03:18:08.546888 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.546868    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-17d8b8-794c9b7f47-pkr5x"]
Sep 14 03:18:08.558185 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.558164    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"]
Sep 14 03:18:08.568090 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:08.568066    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-17d8b8-55d4c96c84-64p96"]
Sep 14 03:18:09.757199 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:09.757168    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17441f5d-c1ae-41ed-acc1-1e3075d91234" path="/var/lib/kubelet/pods/17441f5d-c1ae-41ed-acc1-1e3075d91234/volumes"
Sep 14 03:18:09.757560 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:09.757524    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65bf6740-dca5-4443-8ac1-79500cb5427c" path="/var/lib/kubelet/pods/65bf6740-dca5-4443-8ac1-79500cb5427c/volumes"
Sep 14 03:18:11.123474 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:11.123442    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"]
Sep 14 03:18:11.123944 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:11.123695    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy" containerID="cri-o://c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b" gracePeriod=30
Sep 14 03:18:16.363169 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.363138    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:18:16.487219 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487139    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-data\") pod \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") "
Sep 14 03:18:16.487219 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487178    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-credential-socket\") pod \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") "
Sep 14 03:18:16.487219 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487204    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-envoy\") pod \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") "
Sep 14 03:18:16.487219 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-certs\") pod \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") "
Sep 14 03:18:16.487555 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487258    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-token\") pod \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") "
Sep 14 03:18:16.487555 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487292    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-socket\") pod \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") "
Sep 14 03:18:16.487555 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlgqp\" (UniqueName: \"kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-kube-api-access-xlgqp\") pod \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") "
Sep 14 03:18:16.487555 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-podinfo\") pod \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") "
Sep 14 03:18:16.487555 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487478    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istiod-ca-cert\") pod \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\" (UID: \"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2\") "
Sep 14 03:18:16.487843 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487475    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-data" (OuterVolumeSpecName: "istio-data") pod "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" (UID: "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:16.487843 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" (UID: "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:16.487843 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" (UID: "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:16.487843 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487639    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" (UID: "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:16.487843 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" (UID: "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:18:16.488158 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487891    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:16.488158 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487911    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:16.488158 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487925    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:16.488158 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.487938    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:16.489415 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.489391    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" (UID: "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:16.489597 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.489572    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" (UID: "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 03:18:16.489649 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.489586    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-token" (OuterVolumeSpecName: "istio-token") pod "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" (UID: "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:18:16.489649 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.489606    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-kube-api-access-xlgqp" (OuterVolumeSpecName: "kube-api-access-xlgqp") pod "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" (UID: "cbcd7270-0cdf-4370-b4e9-9b7a36354bc2"). InnerVolumeSpecName "kube-api-access-xlgqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:18:16.565752 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.565727    2580 generic.go:356] "Generic (PLEG): container finished" podID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerID="c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b" exitCode=0
Sep 14 03:18:16.565860 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.565766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" event={"ID":"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2","Type":"ContainerDied","Data":"c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b"}
Sep 14 03:18:16.565860 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.565771    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"
Sep 14 03:18:16.565860 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.565808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs" event={"ID":"cbcd7270-0cdf-4370-b4e9-9b7a36354bc2","Type":"ContainerDied","Data":"18d8ef696abf8b984aa24ed499e2542326bfa2cc8e2ce8e220e062acb27a3132"}
Sep 14 03:18:16.565860 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.565829    2580 scope.go:117] "RemoveContainer" containerID="c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b"
Sep 14 03:18:16.574988 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.574971    2580 scope.go:117] "RemoveContainer" containerID="e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a"
Sep 14 03:18:16.582716 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.582699    2580 scope.go:117] "RemoveContainer" containerID="c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b"
Sep 14 03:18:16.582969 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:18:16.582951    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b\": container with ID starting with c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b not found: ID does not exist" containerID="c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b"
Sep 14 03:18:16.583023 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.582976    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b"} err="failed to get container status \"c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b\": rpc error: code = NotFound desc = could not find container \"c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b\": container with ID starting with c01b93f7f693217398068028fe05e526096c9809d282ff457410697c97be9e7b not found: ID does not exist"
Sep 14 03:18:16.583023 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.582993    2580 scope.go:117] "RemoveContainer" containerID="e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a"
Sep 14 03:18:16.583231 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:18:16.583216    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a\": container with ID starting with e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a not found: ID does not exist" containerID="e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a"
Sep 14 03:18:16.583269 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.583233    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a"} err="failed to get container status \"e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a\": rpc error: code = NotFound desc = could not find container \"e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a\": container with ID starting with e81e5ac7ea66695c9bda3516c4d7d3cf5f14d7d876236047e8f6f847fd6ccb4a not found: ID does not exist"
Sep 14 03:18:16.588802 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.588784    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:16.588802 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.588803    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:16.588929 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.588813    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlgqp\" (UniqueName: \"kubernetes.io/projected/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-kube-api-access-xlgqp\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:16.588929 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.588822    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:16.588929 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.588831    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:16.589133 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.589114    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"]
Sep 14 03:18:16.592824 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.592803    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c9f1e-openshift-default-5cd9c7f58b-qw4qs"]
Sep 14 03:18:16.752837 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:16.752769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:18:17.756595 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:17.756562    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" path="/var/lib/kubelet/pods/cbcd7270-0cdf-4370-b4e9-9b7a36354bc2/volumes"
Sep 14 03:18:31.557967 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:31.557935    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:18:40.501069 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501023    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7674-2385b4"]
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501355    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" containerName="main"
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501366    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" containerName="main"
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501376    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy"
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501382    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy"
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501388    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c500aa2-1218-430f-9e91-0325ec0a08de" containerName="main"
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501394    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c500aa2-1218-430f-9e91-0325ec0a08de" containerName="main"
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501406    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" containerName="main"
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501412    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" containerName="main"
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501418    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy"
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501423    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy"
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501429    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c07fba8-b58c-424d-8861-7da5cce15c8e" containerName="maas-api"
Sep 14 03:18:40.501431 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501434    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c07fba8-b58c-424d-8861-7da5cce15c8e" containerName="maas-api"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501446    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65bf6740-dca5-4443-8ac1-79500cb5427c" containerName="payload-processing"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501451    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65bf6740-dca5-4443-8ac1-79500cb5427c" containerName="payload-processing"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501458    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d873fa5-6e7b-4b32-81b7-35a06abc2d81" containerName="payload-processing"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501463    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d873fa5-6e7b-4b32-81b7-35a06abc2d81" containerName="payload-processing"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501473    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32150566-1bac-4eca-b69f-1b997aeefbbe" containerName="main"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501477    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32150566-1bac-4eca-b69f-1b997aeefbbe" containerName="main"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501482    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07863119-ad92-4bf2-bb9e-8b851262fbe3" containerName="main"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501487    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07863119-ad92-4bf2-bb9e-8b851262fbe3" containerName="main"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501494    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17441f5d-c1ae-41ed-acc1-1e3075d91234" containerName="maas-api"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501499    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="17441f5d-c1ae-41ed-acc1-1e3075d91234" containerName="maas-api"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501551    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c07fba8-b58c-424d-8861-7da5cce15c8e" containerName="maas-api"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501558    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e9616a-a789-4dcf-b4fc-4b6c4b64e06b" containerName="main"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501565    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501570    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d873fa5-6e7b-4b32-81b7-35a06abc2d81" containerName="payload-processing"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501575    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32150566-1bac-4eca-b69f-1b997aeefbbe" containerName="main"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501580    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07863119-ad92-4bf2-bb9e-8b851262fbe3" containerName="main"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501585    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501591    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c500aa2-1218-430f-9e91-0325ec0a08de" containerName="main"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501597    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ee7c3f-7b1e-492e-b507-f5ed6e68d5cf" containerName="main"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501602    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="17441f5d-c1ae-41ed-acc1-1e3075d91234" containerName="maas-api"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501608    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65bf6740-dca5-4443-8ac1-79500cb5427c" containerName="payload-processing"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501614    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy"
Sep 14 03:18:40.501801 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.501619    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy"
Sep 14 03:18:40.506294 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.506275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-2385b4"
Sep 14 03:18:40.508449 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.508427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2qhmh\""
Sep 14 03:18:40.510734 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.510710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-2385b4"]
Sep 14 03:18:40.591642 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.591618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwtn\" (UniqueName: \"kubernetes.io/projected/bcbd20a1-9118-45c3-b376-dfde83d00f88-kube-api-access-dqwtn\") pod \"test-curl-7674-2385b4\" (UID: \"bcbd20a1-9118-45c3-b376-dfde83d00f88\") " pod="opendatahub/test-curl-7674-2385b4"
Sep 14 03:18:40.692827 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.692803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwtn\" (UniqueName: \"kubernetes.io/projected/bcbd20a1-9118-45c3-b376-dfde83d00f88-kube-api-access-dqwtn\") pod \"test-curl-7674-2385b4\" (UID: \"bcbd20a1-9118-45c3-b376-dfde83d00f88\") " pod="opendatahub/test-curl-7674-2385b4"
Sep 14 03:18:40.701625 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.701596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwtn\" (UniqueName: \"kubernetes.io/projected/bcbd20a1-9118-45c3-b376-dfde83d00f88-kube-api-access-dqwtn\") pod \"test-curl-7674-2385b4\" (UID: \"bcbd20a1-9118-45c3-b376-dfde83d00f88\") " pod="opendatahub/test-curl-7674-2385b4"
Sep 14 03:18:40.816829 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:40.816808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-2385b4"
Sep 14 03:18:41.084784 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:41.084758    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-2385b4"]
Sep 14 03:18:41.086597 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:18:41.086573    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcbd20a1_9118_45c3_b376_dfde83d00f88.slice/crio-ccb680924dba71839ed2c89afba2fad3c11c5f4b76367a19e56b01a494ef9606 WatchSource:0}: Error finding container ccb680924dba71839ed2c89afba2fad3c11c5f4b76367a19e56b01a494ef9606: Status 404 returned error can't find the container with id ccb680924dba71839ed2c89afba2fad3c11c5f4b76367a19e56b01a494ef9606
Sep 14 03:18:41.700412 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:41.700380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-2385b4" event={"ID":"bcbd20a1-9118-45c3-b376-dfde83d00f88","Type":"ContainerStarted","Data":"c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31"}
Sep 14 03:18:41.700776 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:41.700423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-2385b4" event={"ID":"bcbd20a1-9118-45c3-b376-dfde83d00f88","Type":"ContainerStarted","Data":"ccb680924dba71839ed2c89afba2fad3c11c5f4b76367a19e56b01a494ef9606"}
Sep 14 03:18:41.716855 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:41.716802    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7674-2385b4" podStartSLOduration=1.716787732 podStartE2EDuration="1.716787732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:18:41.713946506 +0000 UTC m=+1588.439904557" watchObservedRunningTime="2026-09-14 03:18:41.716787732 +0000 UTC m=+1588.442745782"
Sep 14 03:18:42.326072 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:42.326041    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7674-2385b4"]
Sep 14 03:18:42.330246 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:42.330217    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7674-2385b4"]
Sep 14 03:18:42.705716 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:18:42.705691    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7674-2385b4" container="test-curl-7674-2385b4"
Sep 14 03:18:43.360398 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:43.360367    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:18:43.665308 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:43.665224    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"]
Sep 14 03:18:43.710838 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:43.710803    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7674-2385b4" podUID="bcbd20a1-9118-45c3-b376-dfde83d00f88" containerName="test-curl-7674-2385b4" containerID="cri-o://c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31" gracePeriod=2
Sep 14 03:18:43.712655 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:18:43.712629    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-2385b4\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="bcbd20a1-9118-45c3-b376-dfde83d00f88" pod="opendatahub/test-curl-7674-2385b4"
Sep 14 03:18:43.756946 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:18:43.756916    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-2385b4\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="bcbd20a1-9118-45c3-b376-dfde83d00f88" pod="opendatahub/test-curl-7674-2385b4"
Sep 14 03:18:44.744570 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:44.744534    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"]
Sep 14 03:18:45.853201 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:45.853177    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-2385b4"
Sep 14 03:18:45.940802 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:45.940776    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqwtn\" (UniqueName: \"kubernetes.io/projected/bcbd20a1-9118-45c3-b376-dfde83d00f88-kube-api-access-dqwtn\") pod \"bcbd20a1-9118-45c3-b376-dfde83d00f88\" (UID: \"bcbd20a1-9118-45c3-b376-dfde83d00f88\") "
Sep 14 03:18:45.942690 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:45.942666    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbd20a1-9118-45c3-b376-dfde83d00f88-kube-api-access-dqwtn" (OuterVolumeSpecName: "kube-api-access-dqwtn") pod "bcbd20a1-9118-45c3-b376-dfde83d00f88" (UID: "bcbd20a1-9118-45c3-b376-dfde83d00f88"). InnerVolumeSpecName "kube-api-access-dqwtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:18:46.042279 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:46.042252    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqwtn\" (UniqueName: \"kubernetes.io/projected/bcbd20a1-9118-45c3-b376-dfde83d00f88-kube-api-access-dqwtn\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:46.729301 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:46.729269    2580 generic.go:356] "Generic (PLEG): container finished" podID="bcbd20a1-9118-45c3-b376-dfde83d00f88" containerID="c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31" exitCode=137
Sep 14 03:18:46.729467 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:46.729312    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-2385b4"
Sep 14 03:18:46.729467 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:46.729374    2580 scope.go:117] "RemoveContainer" containerID="c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31"
Sep 14 03:18:46.738373 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:46.738356    2580 scope.go:117] "RemoveContainer" containerID="c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31"
Sep 14 03:18:46.738629 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:18:46.738610    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31\": container with ID starting with c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31 not found: ID does not exist" containerID="c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31"
Sep 14 03:18:46.738674 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:46.738637    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31"} err="failed to get container status \"c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31\": rpc error: code = NotFound desc = could not find container \"c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31\": container with ID starting with c21b0c5ffa13734dfa38fce9b36303db806c584da428b49035b28de5f559cf31 not found: ID does not exist"
Sep 14 03:18:46.739357 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:18:46.739337    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-2385b4\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="bcbd20a1-9118-45c3-b376-dfde83d00f88" pod="opendatahub/test-curl-7674-2385b4"
Sep 14 03:18:47.755977 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:47.755945    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcbd20a1-9118-45c3-b376-dfde83d00f88" path="/var/lib/kubelet/pods/bcbd20a1-9118-45c3-b376-dfde83d00f88/volumes"
Sep 14 03:18:53.361909 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:53.361873    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:18:54.075309 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.075280    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"]
Sep 14 03:18:54.075619 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.075608    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy"
Sep 14 03:18:54.075667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.075621    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbcd7270-0cdf-4370-b4e9-9b7a36354bc2" containerName="istio-proxy"
Sep 14 03:18:54.075667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.075630    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcbd20a1-9118-45c3-b376-dfde83d00f88" containerName="test-curl-7674-2385b4"
Sep 14 03:18:54.075667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.075636    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcbd20a1-9118-45c3-b376-dfde83d00f88" containerName="test-curl-7674-2385b4"
Sep 14 03:18:54.075667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.075649    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy"
Sep 14 03:18:54.075667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.075655    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5abadfa-a761-43fe-afe6-9ab492992fb9" containerName="istio-proxy"
Sep 14 03:18:54.075822 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.075713    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcbd20a1-9118-45c3-b376-dfde83d00f88" containerName="test-curl-7674-2385b4"
Sep 14 03:18:54.079993 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.079970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.082754 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.082733    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b32aa\"/\"sub-test-model-2b32aa-kserve-self-signed-certs\""
Sep 14 03:18:54.089201 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.089181    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"]
Sep 14 03:18:54.104367 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.104346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tls-certs\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.104469 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.104382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tmp-dir\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.104469 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.104406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-home\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.104469 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.104444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-model-cache\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.104592 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.104496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-dshm\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.104592 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.104545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57npv\" (UniqueName: \"kubernetes.io/projected/62a0119c-a359-42b4-bdd3-e0e2424a9acc-kube-api-access-57npv\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.205478 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.205447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tls-certs\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.205585 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.205501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tmp-dir\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.205585 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.205528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-home\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.205585 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.205546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-model-cache\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.205585 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.205582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-dshm\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.205813 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.205614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57npv\" (UniqueName: \"kubernetes.io/projected/62a0119c-a359-42b4-bdd3-e0e2424a9acc-kube-api-access-57npv\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.205936 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.205918    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-home\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.206006 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.205986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tmp-dir\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.206067 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.206014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-model-cache\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.207809 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.207784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-dshm\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.207925 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.207908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tls-certs\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.214024 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.214002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57npv\" (UniqueName: \"kubernetes.io/projected/62a0119c-a359-42b4-bdd3-e0e2424a9acc-kube-api-access-57npv\") pod \"sub-test-model-2b32aa-kserve-655659cc-9wxpn\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.390395 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.390319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:18:54.661396 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.661370    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"]
Sep 14 03:18:54.663541 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:18:54.663518    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62a0119c_a359_42b4_bdd3_e0e2424a9acc.slice/crio-33f21bd478a2520777b41afab40823edcd677f46a43d8d1816ccfbda118ead03 WatchSource:0}: Error finding container 33f21bd478a2520777b41afab40823edcd677f46a43d8d1816ccfbda118ead03: Status 404 returned error can't find the container with id 33f21bd478a2520777b41afab40823edcd677f46a43d8d1816ccfbda118ead03
Sep 14 03:18:54.775899 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.775864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn" event={"ID":"62a0119c-a359-42b4-bdd3-e0e2424a9acc","Type":"ContainerStarted","Data":"f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b"}
Sep 14 03:18:54.776064 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:54.775906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn" event={"ID":"62a0119c-a359-42b4-bdd3-e0e2424a9acc","Type":"ContainerStarted","Data":"33f21bd478a2520777b41afab40823edcd677f46a43d8d1816ccfbda118ead03"}
Sep 14 03:18:58.684188 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:58.684147    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb" podUID="80b5a27e-bbb5-4316-87b0-88803a9ffc21" containerName="main" containerID="cri-o://cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05" gracePeriod=45
Sep 14 03:18:58.943335 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:58.943280    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:18:58.963555 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:58.963510    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn" podStartSLOduration=4.9634897460000005 podStartE2EDuration="4.963489746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:18:54.806919018 +0000 UTC m=+1601.532877071" watchObservedRunningTime="2026-09-14 03:18:58.963489746 +0000 UTC m=+1605.689447800"
Sep 14 03:18:59.045282 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045254    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-dshm\") pod \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") "
Sep 14 03:18:59.045442 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-home\") pod \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") "
Sep 14 03:18:59.045442 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045344    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sds6k\" (UniqueName: \"kubernetes.io/projected/80b5a27e-bbb5-4316-87b0-88803a9ffc21-kube-api-access-sds6k\") pod \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") "
Sep 14 03:18:59.045442 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045376    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-model-cache\") pod \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") "
Sep 14 03:18:59.045442 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045402    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tmp-dir\") pod \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") "
Sep 14 03:18:59.045671 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045539    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tls-certs\") pod \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\" (UID: \"80b5a27e-bbb5-4316-87b0-88803a9ffc21\") "
Sep 14 03:18:59.045671 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-model-cache" (OuterVolumeSpecName: "model-cache") pod "80b5a27e-bbb5-4316-87b0-88803a9ffc21" (UID: "80b5a27e-bbb5-4316-87b0-88803a9ffc21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:59.045671 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045659    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80b5a27e-bbb5-4316-87b0-88803a9ffc21" (UID: "80b5a27e-bbb5-4316-87b0-88803a9ffc21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:59.045775 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045636    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-home" (OuterVolumeSpecName: "home") pod "80b5a27e-bbb5-4316-87b0-88803a9ffc21" (UID: "80b5a27e-bbb5-4316-87b0-88803a9ffc21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:59.045866 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045851    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:59.045932 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045869    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:59.045932 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.045878    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:59.047478 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.047454    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-dshm" (OuterVolumeSpecName: "dshm") pod "80b5a27e-bbb5-4316-87b0-88803a9ffc21" (UID: "80b5a27e-bbb5-4316-87b0-88803a9ffc21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:59.047590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.047497    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b5a27e-bbb5-4316-87b0-88803a9ffc21-kube-api-access-sds6k" (OuterVolumeSpecName: "kube-api-access-sds6k") pod "80b5a27e-bbb5-4316-87b0-88803a9ffc21" (UID: "80b5a27e-bbb5-4316-87b0-88803a9ffc21"). InnerVolumeSpecName "kube-api-access-sds6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:18:59.047634 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.047591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80b5a27e-bbb5-4316-87b0-88803a9ffc21" (UID: "80b5a27e-bbb5-4316-87b0-88803a9ffc21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:18:59.147145 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.147119    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sds6k\" (UniqueName: \"kubernetes.io/projected/80b5a27e-bbb5-4316-87b0-88803a9ffc21-kube-api-access-sds6k\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:59.147145 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.147143    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b5a27e-bbb5-4316-87b0-88803a9ffc21-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:59.147305 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.147153    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b5a27e-bbb5-4316-87b0-88803a9ffc21-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:18:59.763619 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.763578    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g" podUID="913fed4a-1b2d-49a7-98c6-f0bb4863308f" containerName="main" containerID="cri-o://19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb" gracePeriod=45
Sep 14 03:18:59.804260 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.804227    2580 generic.go:356] "Generic (PLEG): container finished" podID="80b5a27e-bbb5-4316-87b0-88803a9ffc21" containerID="cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05" exitCode=0
Sep 14 03:18:59.804375 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.804265    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"
Sep 14 03:18:59.804375 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.804272    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb" event={"ID":"80b5a27e-bbb5-4316-87b0-88803a9ffc21","Type":"ContainerDied","Data":"cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05"}
Sep 14 03:18:59.804375 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.804329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb" event={"ID":"80b5a27e-bbb5-4316-87b0-88803a9ffc21","Type":"ContainerDied","Data":"910b1ed83680cb2c775903c7ac59e5572ad224d315915c85948f5f1856dda58a"}
Sep 14 03:18:59.804375 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.804353    2580 scope.go:117] "RemoveContainer" containerID="cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05"
Sep 14 03:18:59.834562 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.834538    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"]
Sep 14 03:18:59.841565 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.841542    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/auth-test-model-2b32aa-kserve-54f6f9967f-98whb"]
Sep 14 03:18:59.874680 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.874657    2580 scope.go:117] "RemoveContainer" containerID="cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05"
Sep 14 03:18:59.875002 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:18:59.874981    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05\": container with ID starting with cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05 not found: ID does not exist" containerID="cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05"
Sep 14 03:18:59.875098 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:18:59.875009    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05"} err="failed to get container status \"cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05\": rpc error: code = NotFound desc = could not find container \"cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05\": container with ID starting with cb370a62873c2b78b9ee95313a81a13e1f8b8e25a6cb1e2a3028377d2352af05 not found: ID does not exist"
Sep 14 03:19:00.012231 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.012208    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:19:00.055916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.055844    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-home\") pod \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") "
Sep 14 03:19:00.055916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.055885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tls-certs\") pod \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") "
Sep 14 03:19:00.056134 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.055925    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-dshm\") pod \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") "
Sep 14 03:19:00.056134 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.055993    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tmp-dir\") pod \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") "
Sep 14 03:19:00.056134 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.056085    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kg7j\" (UniqueName: \"kubernetes.io/projected/913fed4a-1b2d-49a7-98c6-f0bb4863308f-kube-api-access-4kg7j\") pod \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") "
Sep 14 03:19:00.056134 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.056111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-model-cache\") pod \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\" (UID: \"913fed4a-1b2d-49a7-98c6-f0bb4863308f\") "
Sep 14 03:19:00.056358 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.056194    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-home" (OuterVolumeSpecName: "home") pod "913fed4a-1b2d-49a7-98c6-f0bb4863308f" (UID: "913fed4a-1b2d-49a7-98c6-f0bb4863308f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:19:00.056514 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.056488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-model-cache" (OuterVolumeSpecName: "model-cache") pod "913fed4a-1b2d-49a7-98c6-f0bb4863308f" (UID: "913fed4a-1b2d-49a7-98c6-f0bb4863308f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:19:00.056625 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.056519    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:19:00.056625 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.056517    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "913fed4a-1b2d-49a7-98c6-f0bb4863308f" (UID: "913fed4a-1b2d-49a7-98c6-f0bb4863308f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:19:00.058568 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.058538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "913fed4a-1b2d-49a7-98c6-f0bb4863308f" (UID: "913fed4a-1b2d-49a7-98c6-f0bb4863308f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:19:00.058568 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.058553    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-dshm" (OuterVolumeSpecName: "dshm") pod "913fed4a-1b2d-49a7-98c6-f0bb4863308f" (UID: "913fed4a-1b2d-49a7-98c6-f0bb4863308f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:19:00.058710 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.058571    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913fed4a-1b2d-49a7-98c6-f0bb4863308f-kube-api-access-4kg7j" (OuterVolumeSpecName: "kube-api-access-4kg7j") pod "913fed4a-1b2d-49a7-98c6-f0bb4863308f" (UID: "913fed4a-1b2d-49a7-98c6-f0bb4863308f"). InnerVolumeSpecName "kube-api-access-4kg7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:19:00.157406 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.157378    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:19:00.157406 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.157401    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kg7j\" (UniqueName: \"kubernetes.io/projected/913fed4a-1b2d-49a7-98c6-f0bb4863308f-kube-api-access-4kg7j\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:19:00.157562 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.157412    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:19:00.157562 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.157421    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/913fed4a-1b2d-49a7-98c6-f0bb4863308f-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:19:00.157562 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.157428    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/913fed4a-1b2d-49a7-98c6-f0bb4863308f-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:19:00.811343 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.811304    2580 generic.go:356] "Generic (PLEG): container finished" podID="913fed4a-1b2d-49a7-98c6-f0bb4863308f" containerID="19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb" exitCode=0
Sep 14 03:19:00.811343 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.811347    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"
Sep 14 03:19:00.811820 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.811369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g" event={"ID":"913fed4a-1b2d-49a7-98c6-f0bb4863308f","Type":"ContainerDied","Data":"19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb"}
Sep 14 03:19:00.811820 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.811403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g" event={"ID":"913fed4a-1b2d-49a7-98c6-f0bb4863308f","Type":"ContainerDied","Data":"bececed80b95696ba73080e869f04a8ef1536b0c2db8c8aabd8e3d143c5ce060"}
Sep 14 03:19:00.811820 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.811425    2580 scope.go:117] "RemoveContainer" containerID="19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb"
Sep 14 03:19:00.820194 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.820174    2580 scope.go:117] "RemoveContainer" containerID="19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb"
Sep 14 03:19:00.820452 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:19:00.820431    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb\": container with ID starting with 19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb not found: ID does not exist" containerID="19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb"
Sep 14 03:19:00.820511 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.820463    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb"} err="failed to get container status \"19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb\": rpc error: code = NotFound desc = could not find container \"19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb\": container with ID starting with 19042f52531d8575e56c6fee3e69a5f369338e3aa13704ed7018c865d62567fb not found: ID does not exist"
Sep 14 03:19:00.832892 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.832870    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"]
Sep 14 03:19:00.836835 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:00.836812    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/auth-test-model-177593-kserve-5d95c697f9-w5x5g"]
Sep 14 03:19:01.757095 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:01.757061    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80b5a27e-bbb5-4316-87b0-88803a9ffc21" path="/var/lib/kubelet/pods/80b5a27e-bbb5-4316-87b0-88803a9ffc21/volumes"
Sep 14 03:19:01.757461 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:01.757448    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="913fed4a-1b2d-49a7-98c6-f0bb4863308f" path="/var/lib/kubelet/pods/913fed4a-1b2d-49a7-98c6-f0bb4863308f/volumes"
Sep 14 03:19:03.465763 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:03.465726    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:19:04.390658 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:04.390629    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:19:04.390913 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:04.390670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:19:04.403186 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:04.403161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:19:04.844653 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:04.844622    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:19:22.982806 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:22.982756    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:19:47.655679 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:47.655645    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:19:58.064429 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.064400    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7674-065d43"]
Sep 14 03:19:58.064900 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.064734    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b5a27e-bbb5-4316-87b0-88803a9ffc21" containerName="main"
Sep 14 03:19:58.064900 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.064745    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b5a27e-bbb5-4316-87b0-88803a9ffc21" containerName="main"
Sep 14 03:19:58.064900 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.064759    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="913fed4a-1b2d-49a7-98c6-f0bb4863308f" containerName="main"
Sep 14 03:19:58.064900 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.064765    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="913fed4a-1b2d-49a7-98c6-f0bb4863308f" containerName="main"
Sep 14 03:19:58.064900 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.064834    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="913fed4a-1b2d-49a7-98c6-f0bb4863308f" containerName="main"
Sep 14 03:19:58.064900 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.064842    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b5a27e-bbb5-4316-87b0-88803a9ffc21" containerName="main"
Sep 14 03:19:58.067997 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.067980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-065d43"
Sep 14 03:19:58.070205 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.070184    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2qhmh\""
Sep 14 03:19:58.076403 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.076381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-065d43"]
Sep 14 03:19:58.139613 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.139585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwtdv\" (UniqueName: \"kubernetes.io/projected/ab0acff2-8d83-42a6-b977-688e5fc89a48-kube-api-access-mwtdv\") pod \"test-curl-7674-065d43\" (UID: \"ab0acff2-8d83-42a6-b977-688e5fc89a48\") " pod="opendatahub/test-curl-7674-065d43"
Sep 14 03:19:58.240326 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.240288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwtdv\" (UniqueName: \"kubernetes.io/projected/ab0acff2-8d83-42a6-b977-688e5fc89a48-kube-api-access-mwtdv\") pod \"test-curl-7674-065d43\" (UID: \"ab0acff2-8d83-42a6-b977-688e5fc89a48\") " pod="opendatahub/test-curl-7674-065d43"
Sep 14 03:19:58.249692 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.249665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwtdv\" (UniqueName: \"kubernetes.io/projected/ab0acff2-8d83-42a6-b977-688e5fc89a48-kube-api-access-mwtdv\") pod \"test-curl-7674-065d43\" (UID: \"ab0acff2-8d83-42a6-b977-688e5fc89a48\") " pod="opendatahub/test-curl-7674-065d43"
Sep 14 03:19:58.377567 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.377503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-065d43"
Sep 14 03:19:58.666714 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:58.666689    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-065d43"]
Sep 14 03:19:58.669114 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:19:58.669089    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab0acff2_8d83_42a6_b977_688e5fc89a48.slice/crio-eb0b45c6b6ca9cc2a110996957fbe6f6d475c794fa2b7db1e2d187ccd256f4af WatchSource:0}: Error finding container eb0b45c6b6ca9cc2a110996957fbe6f6d475c794fa2b7db1e2d187ccd256f4af: Status 404 returned error can't find the container with id eb0b45c6b6ca9cc2a110996957fbe6f6d475c794fa2b7db1e2d187ccd256f4af
Sep 14 03:19:59.118425 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.118392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-065d43" event={"ID":"ab0acff2-8d83-42a6-b977-688e5fc89a48","Type":"ContainerStarted","Data":"92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d"}
Sep 14 03:19:59.118425 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.118430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-065d43" event={"ID":"ab0acff2-8d83-42a6-b977-688e5fc89a48","Type":"ContainerStarted","Data":"eb0b45c6b6ca9cc2a110996957fbe6f6d475c794fa2b7db1e2d187ccd256f4af"}
Sep 14 03:19:59.136085 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.136024    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7674-065d43" podStartSLOduration=1.136010759 podStartE2EDuration="1.136010759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:19:59.134416756 +0000 UTC m=+1665.860374809" watchObservedRunningTime="2026-09-14 03:19:59.136010759 +0000 UTC m=+1665.861968842"
Sep 14 03:19:59.719914 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.719878    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7674-065d43"]
Sep 14 03:19:59.726412 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.726382    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7674-065d43"]
Sep 14 03:19:59.841274 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.841199    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7674-2fa80f"]
Sep 14 03:19:59.841530 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.841517    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab0acff2-8d83-42a6-b977-688e5fc89a48" containerName="test-curl-7674-065d43"
Sep 14 03:19:59.841586 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.841531    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab0acff2-8d83-42a6-b977-688e5fc89a48" containerName="test-curl-7674-065d43"
Sep 14 03:19:59.841624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.841596    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab0acff2-8d83-42a6-b977-688e5fc89a48" containerName="test-curl-7674-065d43"
Sep 14 03:19:59.844557 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.844541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-2fa80f"
Sep 14 03:19:59.849414 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.849387    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-2fa80f"]
Sep 14 03:19:59.953387 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:19:59.953359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jl22\" (UniqueName: \"kubernetes.io/projected/8e1a9ea9-5915-4167-9c42-22c2480a1705-kube-api-access-9jl22\") pod \"test-curl-7674-2fa80f\" (UID: \"8e1a9ea9-5915-4167-9c42-22c2480a1705\") " pod="opendatahub/test-curl-7674-2fa80f"
Sep 14 03:20:00.054086 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:00.054049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jl22\" (UniqueName: \"kubernetes.io/projected/8e1a9ea9-5915-4167-9c42-22c2480a1705-kube-api-access-9jl22\") pod \"test-curl-7674-2fa80f\" (UID: \"8e1a9ea9-5915-4167-9c42-22c2480a1705\") " pod="opendatahub/test-curl-7674-2fa80f"
Sep 14 03:20:00.062657 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:00.062626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jl22\" (UniqueName: \"kubernetes.io/projected/8e1a9ea9-5915-4167-9c42-22c2480a1705-kube-api-access-9jl22\") pod \"test-curl-7674-2fa80f\" (UID: \"8e1a9ea9-5915-4167-9c42-22c2480a1705\") " pod="opendatahub/test-curl-7674-2fa80f"
Sep 14 03:20:00.123660 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:20:00.123593    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7674-065d43" container="test-curl-7674-065d43"
Sep 14 03:20:00.154828 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:00.154804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-2fa80f"
Sep 14 03:20:00.423672 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:00.423642    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-2fa80f"]
Sep 14 03:20:00.425046 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:20:00.425009    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e1a9ea9_5915_4167_9c42_22c2480a1705.slice/crio-4a29a52940cb2bf5f9ffa897752897435e7334bceae3e105aeada73a13cda63d WatchSource:0}: Error finding container 4a29a52940cb2bf5f9ffa897752897435e7334bceae3e105aeada73a13cda63d: Status 404 returned error can't find the container with id 4a29a52940cb2bf5f9ffa897752897435e7334bceae3e105aeada73a13cda63d
Sep 14 03:20:01.130604 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:01.130564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-2fa80f" event={"ID":"8e1a9ea9-5915-4167-9c42-22c2480a1705","Type":"ContainerStarted","Data":"6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21"}
Sep 14 03:20:01.131045 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:01.130613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-2fa80f" event={"ID":"8e1a9ea9-5915-4167-9c42-22c2480a1705","Type":"ContainerStarted","Data":"4a29a52940cb2bf5f9ffa897752897435e7334bceae3e105aeada73a13cda63d"}
Sep 14 03:20:01.131045 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:01.130723    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7674-065d43" podUID="ab0acff2-8d83-42a6-b977-688e5fc89a48" containerName="test-curl-7674-065d43" containerID="cri-o://92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d" gracePeriod=2
Sep 14 03:20:01.145768 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:01.145727    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7674-2fa80f" podStartSLOduration=2.145713415 podStartE2EDuration="2.145713415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:20:01.145655134 +0000 UTC m=+1667.871613185" watchObservedRunningTime="2026-09-14 03:20:01.145713415 +0000 UTC m=+1667.871671466"
Sep 14 03:20:01.764815 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:01.764782    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7674-2fa80f"]
Sep 14 03:20:01.769359 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:01.769335    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7674-2fa80f"]
Sep 14 03:20:02.135817 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:20:02.135789    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7674-2fa80f" container="test-curl-7674-2fa80f"
Sep 14 03:20:02.595045 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:02.595003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:20:03.141016 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:03.140983    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7674-2fa80f" podUID="8e1a9ea9-5915-4167-9c42-22c2480a1705" containerName="test-curl-7674-2fa80f" containerID="cri-o://6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21" gracePeriod=2
Sep 14 03:20:03.143074 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:20:03.143022    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-2fa80f\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="8e1a9ea9-5915-4167-9c42-22c2480a1705" pod="opendatahub/test-curl-7674-2fa80f"
Sep 14 03:20:03.309848 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:03.309822    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-065d43"
Sep 14 03:20:03.311823 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:20:03.311796    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-2fa80f\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="8e1a9ea9-5915-4167-9c42-22c2480a1705" pod="opendatahub/test-curl-7674-2fa80f"
Sep 14 03:20:03.384384 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:03.384305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwtdv\" (UniqueName: \"kubernetes.io/projected/ab0acff2-8d83-42a6-b977-688e5fc89a48-kube-api-access-mwtdv\") pod \"ab0acff2-8d83-42a6-b977-688e5fc89a48\" (UID: \"ab0acff2-8d83-42a6-b977-688e5fc89a48\") "
Sep 14 03:20:03.386361 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:03.386336    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0acff2-8d83-42a6-b977-688e5fc89a48-kube-api-access-mwtdv" (OuterVolumeSpecName: "kube-api-access-mwtdv") pod "ab0acff2-8d83-42a6-b977-688e5fc89a48" (UID: "ab0acff2-8d83-42a6-b977-688e5fc89a48"). InnerVolumeSpecName "kube-api-access-mwtdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:20:03.485982 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:03.485948    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwtdv\" (UniqueName: \"kubernetes.io/projected/ab0acff2-8d83-42a6-b977-688e5fc89a48-kube-api-access-mwtdv\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:20:03.756085 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:20:03.756057    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-2fa80f\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="8e1a9ea9-5915-4167-9c42-22c2480a1705" pod="opendatahub/test-curl-7674-2fa80f"
Sep 14 03:20:03.757180 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:03.757160    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab0acff2-8d83-42a6-b977-688e5fc89a48" path="/var/lib/kubelet/pods/ab0acff2-8d83-42a6-b977-688e5fc89a48/volumes"
Sep 14 03:20:03.821168 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:03.821134    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"]
Sep 14 03:20:04.147224 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:04.147145    2580 generic.go:356] "Generic (PLEG): container finished" podID="ab0acff2-8d83-42a6-b977-688e5fc89a48" containerID="92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d" exitCode=137
Sep 14 03:20:04.147224 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:04.147200    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-065d43"
Sep 14 03:20:04.147660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:04.147215    2580 scope.go:117] "RemoveContainer" containerID="92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d"
Sep 14 03:20:04.155937 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:04.155920    2580 scope.go:117] "RemoveContainer" containerID="92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d"
Sep 14 03:20:04.156281 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:20:04.156256    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d\": container with ID starting with 92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d not found: ID does not exist" containerID="92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d"
Sep 14 03:20:04.156385 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:04.156282    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d"} err="failed to get container status \"92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d\": rpc error: code = NotFound desc = could not find container \"92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d\": container with ID starting with 92396831043b161b550c356417e9acb059fa6747626bad347cef403c38ceb92d not found: ID does not exist"
Sep 14 03:20:05.317063 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:05.317018    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-2fa80f"
Sep 14 03:20:05.407043 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:05.406953    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jl22\" (UniqueName: \"kubernetes.io/projected/8e1a9ea9-5915-4167-9c42-22c2480a1705-kube-api-access-9jl22\") pod \"8e1a9ea9-5915-4167-9c42-22c2480a1705\" (UID: \"8e1a9ea9-5915-4167-9c42-22c2480a1705\") "
Sep 14 03:20:05.409035 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:05.408999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1a9ea9-5915-4167-9c42-22c2480a1705-kube-api-access-9jl22" (OuterVolumeSpecName: "kube-api-access-9jl22") pod "8e1a9ea9-5915-4167-9c42-22c2480a1705" (UID: "8e1a9ea9-5915-4167-9c42-22c2480a1705"). InnerVolumeSpecName "kube-api-access-9jl22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:20:05.508451 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:05.508417    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jl22\" (UniqueName: \"kubernetes.io/projected/8e1a9ea9-5915-4167-9c42-22c2480a1705-kube-api-access-9jl22\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:20:05.757176 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:05.757152    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e1a9ea9-5915-4167-9c42-22c2480a1705" path="/var/lib/kubelet/pods/8e1a9ea9-5915-4167-9c42-22c2480a1705/volumes"
Sep 14 03:20:06.159502 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:06.159406    2580 generic.go:356] "Generic (PLEG): container finished" podID="8e1a9ea9-5915-4167-9c42-22c2480a1705" containerID="6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21" exitCode=137
Sep 14 03:20:06.159502 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:06.159451    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-2fa80f"
Sep 14 03:20:06.159502 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:06.159498    2580 scope.go:117] "RemoveContainer" containerID="6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21"
Sep 14 03:20:06.167888 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:06.167864    2580 scope.go:117] "RemoveContainer" containerID="6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21"
Sep 14 03:20:06.168149 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:20:06.168130    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21\": container with ID starting with 6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21 not found: ID does not exist" containerID="6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21"
Sep 14 03:20:06.168207 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:06.168160    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21"} err="failed to get container status \"6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21\": rpc error: code = NotFound desc = could not find container \"6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21\": container with ID starting with 6a7af09ac02b241d617269cfbfa6948b1a02d6a966175cd39be85a330593cc21 not found: ID does not exist"
Sep 14 03:20:12.491676 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:12.491639    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:20:18.839421 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:18.839378    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn" podUID="62a0119c-a359-42b4-bdd3-e0e2424a9acc" containerName="main" containerID="cri-o://f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b" gracePeriod=45
Sep 14 03:20:19.090248 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.090192    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:20:19.225797 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.225767    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tls-certs\") pod \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") "
Sep 14 03:20:19.225969 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.225819    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57npv\" (UniqueName: \"kubernetes.io/projected/62a0119c-a359-42b4-bdd3-e0e2424a9acc-kube-api-access-57npv\") pod \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") "
Sep 14 03:20:19.225969 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.225846    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-dshm\") pod \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") "
Sep 14 03:20:19.225969 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.225954    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tmp-dir\") pod \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") "
Sep 14 03:20:19.226189 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.226015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-model-cache\") pod \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") "
Sep 14 03:20:19.226189 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.226088    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-home\") pod \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\" (UID: \"62a0119c-a359-42b4-bdd3-e0e2424a9acc\") "
Sep 14 03:20:19.226298 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.226260    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62a0119c-a359-42b4-bdd3-e0e2424a9acc" (UID: "62a0119c-a359-42b4-bdd3-e0e2424a9acc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:20:19.226298 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.226265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-model-cache" (OuterVolumeSpecName: "model-cache") pod "62a0119c-a359-42b4-bdd3-e0e2424a9acc" (UID: "62a0119c-a359-42b4-bdd3-e0e2424a9acc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:20:19.226429 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.226406    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-home" (OuterVolumeSpecName: "home") pod "62a0119c-a359-42b4-bdd3-e0e2424a9acc" (UID: "62a0119c-a359-42b4-bdd3-e0e2424a9acc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:20:19.226490 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.226449    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:20:19.226490 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.226467    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:20:19.228127 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.228105    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-dshm" (OuterVolumeSpecName: "dshm") pod "62a0119c-a359-42b4-bdd3-e0e2424a9acc" (UID: "62a0119c-a359-42b4-bdd3-e0e2424a9acc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:20:19.228127 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.228113    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62a0119c-a359-42b4-bdd3-e0e2424a9acc" (UID: "62a0119c-a359-42b4-bdd3-e0e2424a9acc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:20:19.228260 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.228218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a0119c-a359-42b4-bdd3-e0e2424a9acc-kube-api-access-57npv" (OuterVolumeSpecName: "kube-api-access-57npv") pod "62a0119c-a359-42b4-bdd3-e0e2424a9acc" (UID: "62a0119c-a359-42b4-bdd3-e0e2424a9acc"). InnerVolumeSpecName "kube-api-access-57npv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:20:19.236397 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.236375    2580 generic.go:356] "Generic (PLEG): container finished" podID="62a0119c-a359-42b4-bdd3-e0e2424a9acc" containerID="f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b" exitCode=0
Sep 14 03:20:19.236482 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.236426    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"
Sep 14 03:20:19.236539 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.236429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn" event={"ID":"62a0119c-a359-42b4-bdd3-e0e2424a9acc","Type":"ContainerDied","Data":"f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b"}
Sep 14 03:20:19.236600 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.236546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn" event={"ID":"62a0119c-a359-42b4-bdd3-e0e2424a9acc","Type":"ContainerDied","Data":"33f21bd478a2520777b41afab40823edcd677f46a43d8d1816ccfbda118ead03"}
Sep 14 03:20:19.236600 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.236579    2580 scope.go:117] "RemoveContainer" containerID="f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b"
Sep 14 03:20:19.248906 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.248892    2580 scope.go:117] "RemoveContainer" containerID="f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b"
Sep 14 03:20:19.249197 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:20:19.249177    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b\": container with ID starting with f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b not found: ID does not exist" containerID="f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b"
Sep 14 03:20:19.249255 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.249203    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b"} err="failed to get container status \"f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b\": rpc error: code = NotFound desc = could not find container \"f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b\": container with ID starting with f1aa80228ebdc3d26f06efe65a90d39f84968ef4fc56b79b89518e7bb72b217b not found: ID does not exist"
Sep 14 03:20:19.261092 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.261071    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"]
Sep 14 03:20:19.267316 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.267293    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/sub-test-model-2b32aa-kserve-655659cc-9wxpn"]
Sep 14 03:20:19.326866 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.326841    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57npv\" (UniqueName: \"kubernetes.io/projected/62a0119c-a359-42b4-bdd3-e0e2424a9acc-kube-api-access-57npv\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:20:19.326866 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.326863    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:20:19.326989 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.326873    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a0119c-a359-42b4-bdd3-e0e2424a9acc-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:20:19.326989 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.326881    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a0119c-a359-42b4-bdd3-e0e2424a9acc-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:20:19.756834 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:19.756801    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a0119c-a359-42b4-bdd3-e0e2424a9acc" path="/var/lib/kubelet/pods/62a0119c-a359-42b4-bdd3-e0e2424a9acc/volumes"
Sep 14 03:20:27.789660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.789629    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"]
Sep 14 03:20:27.790009 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.789958    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1a9ea9-5915-4167-9c42-22c2480a1705" containerName="test-curl-7674-2fa80f"
Sep 14 03:20:27.790009 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.789969    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1a9ea9-5915-4167-9c42-22c2480a1705" containerName="test-curl-7674-2fa80f"
Sep 14 03:20:27.790009 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.789989    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a0119c-a359-42b4-bdd3-e0e2424a9acc" containerName="main"
Sep 14 03:20:27.790009 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.789996    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a0119c-a359-42b4-bdd3-e0e2424a9acc" containerName="main"
Sep 14 03:20:27.790180 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.790084    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a0119c-a359-42b4-bdd3-e0e2424a9acc" containerName="main"
Sep 14 03:20:27.790180 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.790092    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1a9ea9-5915-4167-9c42-22c2480a1705" containerName="test-curl-7674-2fa80f"
Sep 14 03:20:27.794416 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.794399    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.798006 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.797980    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b32aa\"/\"kube-root-ca.crt\""
Sep 14 03:20:27.798330 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.798308    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b32aa\"/\"default-dockercfg-rjpdw\""
Sep 14 03:20:27.798460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.798312    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b32aa\"/\"openshift-service-ca.crt\""
Sep 14 03:20:27.798528 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.798456    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b32aa\"/\"rate-test-model-2b32aa-kserve-self-signed-certs\""
Sep 14 03:20:27.813653 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.813630    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"]
Sep 14 03:20:27.898812 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.898786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-model-cache\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.898956 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.898839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-tmp-dir\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.898956 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.898861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052bbddf-df52-4339-9016-f12dd033a73c-tls-certs\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.899106 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.898948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvk5t\" (UniqueName: \"kubernetes.io/projected/052bbddf-df52-4339-9016-f12dd033a73c-kube-api-access-bvk5t\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.899106 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.899012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-home\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.899106 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.899076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-dshm\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.999715 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.999679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-home\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.999916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.999727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-dshm\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.999916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.999772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-model-cache\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.999916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.999797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-tmp-dir\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.999916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.999812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052bbddf-df52-4339-9016-f12dd033a73c-tls-certs\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:27.999916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:27.999902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvk5t\" (UniqueName: \"kubernetes.io/projected/052bbddf-df52-4339-9016-f12dd033a73c-kube-api-access-bvk5t\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:28.000235 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:28.000196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-home\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:28.000280 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:28.000237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-model-cache\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:28.000280 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:28.000257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-tmp-dir\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:28.002060 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:28.002006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-dshm\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:28.002302 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:28.002286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052bbddf-df52-4339-9016-f12dd033a73c-tls-certs\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:28.008792 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:28.008769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvk5t\" (UniqueName: \"kubernetes.io/projected/052bbddf-df52-4339-9016-f12dd033a73c-kube-api-access-bvk5t\") pod \"rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") " pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:28.103276 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:28.103214    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:28.387022 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:28.384137    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"]
Sep 14 03:20:28.388510 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:20:28.388479    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod052bbddf_df52_4339_9016_f12dd033a73c.slice/crio-da229551b8df90d3c969e2278e90ef7589e37789ce5af769b2a229fdc8c4f723 WatchSource:0}: Error finding container da229551b8df90d3c969e2278e90ef7589e37789ce5af769b2a229fdc8c4f723: Status 404 returned error can't find the container with id da229551b8df90d3c969e2278e90ef7589e37789ce5af769b2a229fdc8c4f723
Sep 14 03:20:29.297040 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:29.296986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2" event={"ID":"052bbddf-df52-4339-9016-f12dd033a73c","Type":"ContainerStarted","Data":"7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e"}
Sep 14 03:20:29.297470 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:29.297059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2" event={"ID":"052bbddf-df52-4339-9016-f12dd033a73c","Type":"ContainerStarted","Data":"da229551b8df90d3c969e2278e90ef7589e37789ce5af769b2a229fdc8c4f723"}
Sep 14 03:20:29.315901 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:29.315847    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2" podStartSLOduration=2.315832456 podStartE2EDuration="2.315832456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:20:29.313839041 +0000 UTC m=+1696.039797096" watchObservedRunningTime="2026-09-14 03:20:29.315832456 +0000 UTC m=+1696.041790508"
Sep 14 03:20:30.754510 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:30.754475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:20:38.103668 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:38.103584    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:38.103668 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:38.103639    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:38.115606 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:38.115584    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:38.352749 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:38.352719    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:20:45.660572 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.660539    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"]
Sep 14 03:20:45.663885 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.663861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.683841 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.683811    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"default-dockercfg-54rpf\""
Sep 14 03:20:45.684042 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.683810    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"openshift-service-ca.crt\""
Sep 14 03:20:45.684042 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.683821    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"rate-test-model-177593-kserve-self-signed-certs\""
Sep 14 03:20:45.684232 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.684214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"kube-root-ca.crt\""
Sep 14 03:20:45.686472 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.686451    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"]
Sep 14 03:20:45.749940 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.749919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-model-cache\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.750059 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.749947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tls-certs\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.750059 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.750005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqqnc\" (UniqueName: \"kubernetes.io/projected/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-kube-api-access-rqqnc\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.750153 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.750061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tmp-dir\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.750153 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.750088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-dshm\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.750153 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.750108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-home\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.851380 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.851351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-model-cache\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.851380 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.851382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tls-certs\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.851584 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.851446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqqnc\" (UniqueName: \"kubernetes.io/projected/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-kube-api-access-rqqnc\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.851584 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.851471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tmp-dir\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.851584 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.851490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-dshm\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.851584 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.851528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-home\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.851952 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.851930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-home\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.852021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.851975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-model-cache\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.852021 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.852008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tmp-dir\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.853806 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.853776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-dshm\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.854054 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.854023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tls-certs\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.859474 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.859450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqqnc\" (UniqueName: \"kubernetes.io/projected/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-kube-api-access-rqqnc\") pod \"rate-test-model-177593-kserve-687db4bf9-ttwz7\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") " pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:45.974263 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:45.974200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:46.244966 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:46.244938    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"]
Sep 14 03:20:46.246469 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:20:46.246442    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod328cd6b1_1bff_4f0f_8baa_dc5bab8cf502.slice/crio-c2adac7970f451bfc81f092b607dd8a697c096a787d05cb5a96adf7c9034d8d4 WatchSource:0}: Error finding container c2adac7970f451bfc81f092b607dd8a697c096a787d05cb5a96adf7c9034d8d4: Status 404 returned error can't find the container with id c2adac7970f451bfc81f092b607dd8a697c096a787d05cb5a96adf7c9034d8d4
Sep 14 03:20:46.384520 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:46.384483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7" event={"ID":"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502","Type":"ContainerStarted","Data":"40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3"}
Sep 14 03:20:46.384520 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:46.384522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7" event={"ID":"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502","Type":"ContainerStarted","Data":"c2adac7970f451bfc81f092b607dd8a697c096a787d05cb5a96adf7c9034d8d4"}
Sep 14 03:20:46.403830 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:46.403782    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7" podStartSLOduration=1.403767284 podStartE2EDuration="1.403767284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:20:46.401280794 +0000 UTC m=+1713.127238859" watchObservedRunningTime="2026-09-14 03:20:46.403767284 +0000 UTC m=+1713.129725332"
Sep 14 03:20:49.161095 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:49.161060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:20:55.974779 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:55.974730    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:55.974779 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:55.974789    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:55.987531 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:55.987506    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:20:56.449864 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:20:56.449833    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:21:07.661047 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:07.661002    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:21:14.121546 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:14.121514    2580 scope.go:117] "RemoveContainer" containerID="2b42943e041d868b685a7d45aec07f844b6fa4e4efae7dc38bce20a540506ce4"
Sep 14 03:21:17.107158 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:17.107126    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:21:23.916434 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:23.916398    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"]
Sep 14 03:21:24.706668 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:24.706631    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"]
Sep 14 03:21:25.446767 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.446729    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"]
Sep 14 03:21:25.450447 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.450429    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.453274 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.453256    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d081a3-openshift-default-dockercfg-gb4vk\""
Sep 14 03:21:25.465711 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.465687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"]
Sep 14 03:21:25.490836 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.490809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt424\" (UniqueName: \"kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-kube-api-access-wt424\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.490953 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.490859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-socket\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.490953 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.490888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istiod-ca-cert\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.490953 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.490929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-credential-socket\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.491119 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.491079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-certs\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.491119 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.491108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-envoy\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.491209 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.491139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-token\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.491209 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.491175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-podinfo\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.491294 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.491208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-data\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.592359 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt424\" (UniqueName: \"kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-kube-api-access-wt424\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.592359 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-socket\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.592617 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istiod-ca-cert\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.592617 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-credential-socket\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.592766 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-certs\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.592766 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-envoy\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.592766 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-token\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.592934 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-podinfo\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.592934 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-socket\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.592934 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-data\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.592934 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592918    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-credential-socket\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.593192 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.592982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-certs\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.593192 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.593138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istiod-ca-cert\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.593306 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.593224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-data\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.595153 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.595126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-envoy\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.595596 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.595574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-podinfo\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.601317 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.601291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-token\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.601586 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.601567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt424\" (UniqueName: \"kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-kube-api-access-wt424\") pod \"e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") " pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:25.760545 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:25.760521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:26.044277 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:26.042612    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"]
Sep 14 03:21:26.049175 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:26.049137    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:21:26.049299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:26.049203    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:21:26.049299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:26.049231    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:21:26.602974 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:26.602933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" event={"ID":"b0f64539-7c06-43b8-9b0a-d875804e9e9b","Type":"ContainerStarted","Data":"600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967"}
Sep 14 03:21:26.603439 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:26.602991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" event={"ID":"b0f64539-7c06-43b8-9b0a-d875804e9e9b","Type":"ContainerStarted","Data":"27bae8c8fc7924a91550486171e37558bc4de95a00cd14402dc8b82f7eaefebd"}
Sep 14 03:21:26.624533 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:26.624481    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" podStartSLOduration=1.624464014 podStartE2EDuration="1.624464014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:21:26.622161668 +0000 UTC m=+1753.348119736" watchObservedRunningTime="2026-09-14 03:21:26.624464014 +0000 UTC m=+1753.350422131"
Sep 14 03:21:26.761043 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:26.761004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:27.185272 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:27.185237    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:21:27.765924 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:27.765893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:28.614640 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:28.614608    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:28.615476 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:28.615456    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:31.218883 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.218852    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"]
Sep 14 03:21:31.222713 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.222693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:21:31.227573 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.227553    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d081a3-dockercfg-46cdr\""
Sep 14 03:21:31.227985 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.227969    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d081a3\""
Sep 14 03:21:31.239000 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.238978    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"]
Sep 14 03:21:31.245053 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.245009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36f749f-26a5-4a01-9693-14b4a07e6906-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l\" (UID: \"f36f749f-26a5-4a01-9693-14b4a07e6906\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:21:31.245207 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.245187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jxj\" (UniqueName: \"kubernetes.io/projected/f36f749f-26a5-4a01-9693-14b4a07e6906-kube-api-access-z8jxj\") pod \"payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l\" (UID: \"f36f749f-26a5-4a01-9693-14b4a07e6906\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:21:31.346022 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.345980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jxj\" (UniqueName: \"kubernetes.io/projected/f36f749f-26a5-4a01-9693-14b4a07e6906-kube-api-access-z8jxj\") pod \"payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l\" (UID: \"f36f749f-26a5-4a01-9693-14b4a07e6906\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:21:31.346229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.346062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36f749f-26a5-4a01-9693-14b4a07e6906-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l\" (UID: \"f36f749f-26a5-4a01-9693-14b4a07e6906\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:21:31.346768 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.346746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36f749f-26a5-4a01-9693-14b4a07e6906-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l\" (UID: \"f36f749f-26a5-4a01-9693-14b4a07e6906\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:21:31.358905 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.358874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jxj\" (UniqueName: \"kubernetes.io/projected/f36f749f-26a5-4a01-9693-14b4a07e6906-kube-api-access-z8jxj\") pod \"payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l\" (UID: \"f36f749f-26a5-4a01-9693-14b4a07e6906\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:21:31.533890 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.533857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:21:31.811499 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:31.811468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"]
Sep 14 03:21:31.812605 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:21:31.812576    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36f749f_26a5_4a01_9693_14b4a07e6906.slice/crio-0313c9240432e928122882b22ebaa232eb0a8f33e5ea8383fe0638332a2543ce WatchSource:0}: Error finding container 0313c9240432e928122882b22ebaa232eb0a8f33e5ea8383fe0638332a2543ce: Status 404 returned error can't find the container with id 0313c9240432e928122882b22ebaa232eb0a8f33e5ea8383fe0638332a2543ce
Sep 14 03:21:32.637330 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:32.637288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l" event={"ID":"f36f749f-26a5-4a01-9693-14b4a07e6906","Type":"ContainerStarted","Data":"ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad"}
Sep 14 03:21:32.637745 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:32.637361    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:21:32.637745 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:32.637389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l" event={"ID":"f36f749f-26a5-4a01-9693-14b4a07e6906","Type":"ContainerStarted","Data":"0313c9240432e928122882b22ebaa232eb0a8f33e5ea8383fe0638332a2543ce"}
Sep 14 03:21:32.659858 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:32.659810    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l" podStartSLOduration=1.659796827 podStartE2EDuration="1.659796827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:21:32.656545799 +0000 UTC m=+1759.382503851" watchObservedRunningTime="2026-09-14 03:21:32.659796827 +0000 UTC m=+1759.385754879"
Sep 14 03:21:38.934961 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:38.934921    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2" podUID="052bbddf-df52-4339-9016-f12dd033a73c" containerName="main" containerID="cri-o://7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e" gracePeriod=45
Sep 14 03:21:39.228223 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.228192    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:21:39.327510 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.327477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvk5t\" (UniqueName: \"kubernetes.io/projected/052bbddf-df52-4339-9016-f12dd033a73c-kube-api-access-bvk5t\") pod \"052bbddf-df52-4339-9016-f12dd033a73c\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") "
Sep 14 03:21:39.327686 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.327544    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-tmp-dir\") pod \"052bbddf-df52-4339-9016-f12dd033a73c\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") "
Sep 14 03:21:39.327686 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.327646    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052bbddf-df52-4339-9016-f12dd033a73c-tls-certs\") pod \"052bbddf-df52-4339-9016-f12dd033a73c\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") "
Sep 14 03:21:39.327838 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.327689    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-home\") pod \"052bbddf-df52-4339-9016-f12dd033a73c\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") "
Sep 14 03:21:39.327838 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.327735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-model-cache\") pod \"052bbddf-df52-4339-9016-f12dd033a73c\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") "
Sep 14 03:21:39.327838 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.327759    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-dshm\") pod \"052bbddf-df52-4339-9016-f12dd033a73c\" (UID: \"052bbddf-df52-4339-9016-f12dd033a73c\") "
Sep 14 03:21:39.327838 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.327774    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "052bbddf-df52-4339-9016-f12dd033a73c" (UID: "052bbddf-df52-4339-9016-f12dd033a73c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:21:39.328086 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.327963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-model-cache" (OuterVolumeSpecName: "model-cache") pod "052bbddf-df52-4339-9016-f12dd033a73c" (UID: "052bbddf-df52-4339-9016-f12dd033a73c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:21:39.328086 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.327978    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-home" (OuterVolumeSpecName: "home") pod "052bbddf-df52-4339-9016-f12dd033a73c" (UID: "052bbddf-df52-4339-9016-f12dd033a73c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:21:39.328195 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.328169    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:39.328195 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.328190    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:39.328285 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.328206    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:39.329811 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.329786    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052bbddf-df52-4339-9016-f12dd033a73c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "052bbddf-df52-4339-9016-f12dd033a73c" (UID: "052bbddf-df52-4339-9016-f12dd033a73c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:21:39.329811 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.329801    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052bbddf-df52-4339-9016-f12dd033a73c-kube-api-access-bvk5t" (OuterVolumeSpecName: "kube-api-access-bvk5t") pod "052bbddf-df52-4339-9016-f12dd033a73c" (UID: "052bbddf-df52-4339-9016-f12dd033a73c"). InnerVolumeSpecName "kube-api-access-bvk5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:21:39.329964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.329857    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-dshm" (OuterVolumeSpecName: "dshm") pod "052bbddf-df52-4339-9016-f12dd033a73c" (UID: "052bbddf-df52-4339-9016-f12dd033a73c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:21:39.429241 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.429209    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052bbddf-df52-4339-9016-f12dd033a73c-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:39.429241 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.429235    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052bbddf-df52-4339-9016-f12dd033a73c-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:39.429241 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.429247    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvk5t\" (UniqueName: \"kubernetes.io/projected/052bbddf-df52-4339-9016-f12dd033a73c-kube-api-access-bvk5t\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:39.676684 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.676651    2580 generic.go:356] "Generic (PLEG): container finished" podID="052bbddf-df52-4339-9016-f12dd033a73c" containerID="7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e" exitCode=0
Sep 14 03:21:39.676871 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.676689    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"
Sep 14 03:21:39.676871 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.676731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2" event={"ID":"052bbddf-df52-4339-9016-f12dd033a73c","Type":"ContainerDied","Data":"7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e"}
Sep 14 03:21:39.676871 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.676777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2" event={"ID":"052bbddf-df52-4339-9016-f12dd033a73c","Type":"ContainerDied","Data":"da229551b8df90d3c969e2278e90ef7589e37789ce5af769b2a229fdc8c4f723"}
Sep 14 03:21:39.676871 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.676799    2580 scope.go:117] "RemoveContainer" containerID="7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e"
Sep 14 03:21:39.686884 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.686869    2580 scope.go:117] "RemoveContainer" containerID="7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e"
Sep 14 03:21:39.687154 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:21:39.687137    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e\": container with ID starting with 7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e not found: ID does not exist" containerID="7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e"
Sep 14 03:21:39.687201 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.687162    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e"} err="failed to get container status \"7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e\": rpc error: code = NotFound desc = could not find container \"7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e\": container with ID starting with 7da8d5b803ffc47bd7ce9645390b7cf2ea591e175a9ba827d6725f5497ea581e not found: ID does not exist"
Sep 14 03:21:39.700160 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.700133    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"]
Sep 14 03:21:39.709449 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.709422    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b32aa/rate-test-model-2b32aa-kserve-7f7c9b9fcf-c7gc2"]
Sep 14 03:21:39.724935 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.724908    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7" podUID="328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" containerName="main" containerID="cri-o://40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3" gracePeriod=45
Sep 14 03:21:39.757730 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.757708    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="052bbddf-df52-4339-9016-f12dd033a73c" path="/var/lib/kubelet/pods/052bbddf-df52-4339-9016-f12dd033a73c/volumes"
Sep 14 03:21:39.764354 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.764327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:21:39.977000 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:39.976976    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:21:40.035532 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.035502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-model-cache\") pod \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") "
Sep 14 03:21:40.035683 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.035563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tmp-dir\") pod \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") "
Sep 14 03:21:40.035683 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.035582    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-home\") pod \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") "
Sep 14 03:21:40.035683 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.035602    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tls-certs\") pod \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") "
Sep 14 03:21:40.035683 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.035677    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqqnc\" (UniqueName: \"kubernetes.io/projected/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-kube-api-access-rqqnc\") pod \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") "
Sep 14 03:21:40.035857 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.035717    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-dshm\") pod \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\" (UID: \"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502\") "
Sep 14 03:21:40.035857 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.035774    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-model-cache" (OuterVolumeSpecName: "model-cache") pod "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" (UID: "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:21:40.035857 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.035784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" (UID: "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:21:40.035857 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.035844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-home" (OuterVolumeSpecName: "home") pod "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" (UID: "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:21:40.036020 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.035984    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:40.036020 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.035996    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:40.036020 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.036005    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:40.037774 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.037739    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-dshm" (OuterVolumeSpecName: "dshm") pod "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" (UID: "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:21:40.037883 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.037834    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" (UID: "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:21:40.037883 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.037855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-kube-api-access-rqqnc" (OuterVolumeSpecName: "kube-api-access-rqqnc") pod "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" (UID: "328cd6b1-1bff-4f0f-8baa-dc5bab8cf502"). InnerVolumeSpecName "kube-api-access-rqqnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:21:40.136649 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.136617    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:40.136649 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.136646    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqqnc\" (UniqueName: \"kubernetes.io/projected/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-kube-api-access-rqqnc\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:40.136649 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.136655    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:21:40.683925 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.683890    2580 generic.go:356] "Generic (PLEG): container finished" podID="328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" containerID="40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3" exitCode=0
Sep 14 03:21:40.684144 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.683930    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"
Sep 14 03:21:40.684144 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.683978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7" event={"ID":"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502","Type":"ContainerDied","Data":"40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3"}
Sep 14 03:21:40.684144 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.684050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7" event={"ID":"328cd6b1-1bff-4f0f-8baa-dc5bab8cf502","Type":"ContainerDied","Data":"c2adac7970f451bfc81f092b607dd8a697c096a787d05cb5a96adf7c9034d8d4"}
Sep 14 03:21:40.684144 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.684073    2580 scope.go:117] "RemoveContainer" containerID="40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3"
Sep 14 03:21:40.698506 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.698467    2580 scope.go:117] "RemoveContainer" containerID="40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3"
Sep 14 03:21:40.698928 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:21:40.698808    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3\": container with ID starting with 40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3 not found: ID does not exist" containerID="40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3"
Sep 14 03:21:40.698928 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.698844    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3"} err="failed to get container status \"40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3\": rpc error: code = NotFound desc = could not find container \"40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3\": container with ID starting with 40592e53ed7625ed4c7369cf5ef86657f1fc626612838002b5a7f9c1ed30d3a3 not found: ID does not exist"
Sep 14 03:21:40.708676 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.708650    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"]
Sep 14 03:21:40.712875 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:40.712500    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/rate-test-model-177593-kserve-687db4bf9-ttwz7"]
Sep 14 03:21:41.763354 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:41.763317    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" path="/var/lib/kubelet/pods/328cd6b1-1bff-4f0f-8baa-dc5bab8cf502/volumes"
Sep 14 03:21:43.644931 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:43.644904    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:21:43.955629 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:43.955599    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": read tcp 10.132.0.2:33040->10.132.0.91:15021: read: connection reset by peer" start-of-body=
Sep 14 03:21:43.955776 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:43.955667    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": read tcp 10.132.0.2:33040->10.132.0.91:15021: read: connection reset by peer"
Sep 14 03:21:44.712292 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:44.712259    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerID="600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967" exitCode=0
Sep 14 03:21:44.712710 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:44.712319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" event={"ID":"b0f64539-7c06-43b8-9b0a-d875804e9e9b","Type":"ContainerDied","Data":"600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967"}
Sep 14 03:21:44.712710 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:44.712664    2580 scope.go:117] "RemoveContainer" containerID="600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967"
Sep 14 03:21:44.713800 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:44.713759    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:21:44.713901 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:44.713834    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:21:44.713901 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:44.713873    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:21:45.723118 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:45.723079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" event={"ID":"b0f64539-7c06-43b8-9b0a-d875804e9e9b","Type":"ContainerStarted","Data":"e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619"}
Sep 14 03:21:45.761623 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:45.761598    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:45.761782 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:45.761746    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 14 03:21:45.761833 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:45.761789    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 14 03:21:46.761402 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:46.761367    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 14 03:21:46.761768 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:46.761416    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 14 03:21:47.761612 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:47.761579    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 14 03:21:47.762101 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:47.761631    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 14 03:21:48.761682 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:48.761644    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 14 03:21:48.762092 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:48.761705    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 14 03:21:49.761097 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:49.761056    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 14 03:21:49.761294 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:49.761131    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 14 03:21:50.765527 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:50.765494    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:50.765958 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:50.765928    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:50.766716 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:50.766691    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:21:54.572244 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.572211    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"]
Sep 14 03:21:54.572596 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.572586    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052bbddf-df52-4339-9016-f12dd033a73c" containerName="main"
Sep 14 03:21:54.572596 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.572597    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="052bbddf-df52-4339-9016-f12dd033a73c" containerName="main"
Sep 14 03:21:54.572672 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.572608    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" containerName="main"
Sep 14 03:21:54.572672 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.572614    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" containerName="main"
Sep 14 03:21:54.572733 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.572674    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="328cd6b1-1bff-4f0f-8baa-dc5bab8cf502" containerName="main"
Sep 14 03:21:54.572733 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.572688    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="052bbddf-df52-4339-9016-f12dd033a73c" containerName="main"
Sep 14 03:21:54.575772 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.575753    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.578139 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.578119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8d29d7-openshift-default-dockercfg-5nnf8\""
Sep 14 03:21:54.591463 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.591442    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"]
Sep 14 03:21:54.672179 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.672144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-podinfo\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.672340 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.672202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-credential-socket\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.672340 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.672219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-envoy\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.672340 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.672242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-token\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.672340 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.672306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-socket\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.672517 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.672354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-data\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.672517 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.672385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdpf5\" (UniqueName: \"kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-kube-api-access-vdpf5\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.672517 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.672409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istiod-ca-cert\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.672517 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.672446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-certs\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773078 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-credential-socket\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773078 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-envoy\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773321 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-token\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773321 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-socket\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773321 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-data\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773321 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdpf5\" (UniqueName: \"kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-kube-api-access-vdpf5\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773321 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istiod-ca-cert\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773584 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-certs\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773584 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-podinfo\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773719 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-credential-socket\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773781 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-certs\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773781 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-socket\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.773915 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.773825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-data\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.774164 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.774147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istiod-ca-cert\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.775392 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.775367    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-envoy\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.775693 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.775676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-podinfo\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.781803 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.781779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-token\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.792096 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.792078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdpf5\" (UniqueName: \"kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-kube-api-access-vdpf5\") pod \"e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") " pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:54.885443 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:54.885378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:55.200071 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:55.200041    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"]
Sep 14 03:21:55.200478 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:21:55.200454    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6a250f_ebe5_4a4c_93c5_6ab9a8d16f58.slice/crio-4ec70f6168b1474724291d661b81cd03847a8ef025c9ffbcb939fc6c5878635a WatchSource:0}: Error finding container 4ec70f6168b1474724291d661b81cd03847a8ef025c9ffbcb939fc6c5878635a: Status 404 returned error can't find the container with id 4ec70f6168b1474724291d661b81cd03847a8ef025c9ffbcb939fc6c5878635a
Sep 14 03:21:55.202637 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:55.202604    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:21:55.202703 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:55.202673    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:21:55.202741 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:55.202712    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:21:55.778514 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:55.778474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" event={"ID":"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58","Type":"ContainerStarted","Data":"3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3"}
Sep 14 03:21:55.779024 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:55.778532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" event={"ID":"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58","Type":"ContainerStarted","Data":"4ec70f6168b1474724291d661b81cd03847a8ef025c9ffbcb939fc6c5878635a"}
Sep 14 03:21:55.801507 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:55.801458    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" podStartSLOduration=1.8014434750000001 podStartE2EDuration="1.801443475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:21:55.799453524 +0000 UTC m=+1782.525411575" watchObservedRunningTime="2026-09-14 03:21:55.801443475 +0000 UTC m=+1782.527401520"
Sep 14 03:21:55.886316 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:55.886295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:56.891189 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:56.891158    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:57.791992 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:57.791960    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:21:57.793024 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:21:57.793003    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:22:00.451698 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.451664    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"]
Sep 14 03:22:00.456862 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.456840    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:00.458728 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.458705    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8d29d7-serving-cert\""
Sep 14 03:22:00.459356 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.459335    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jf78h\""
Sep 14 03:22:00.459413 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.459369    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 03:22:00.459740 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.459720    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 03:22:00.459783 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.459751    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:22:00.466920 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.466896    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"]
Sep 14 03:22:00.520700 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.520668    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0119b689-80c4-414f-8dc1-13b01d0e7c62-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:00.520821 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.520713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:00.520821 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.520764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdr4z\" (UniqueName: \"kubernetes.io/projected/0119b689-80c4-414f-8dc1-13b01d0e7c62-kube-api-access-xdr4z\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:00.520821 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.520797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-metrics-certs\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:00.621661 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.621624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-metrics-certs\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:00.621841 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.621715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0119b689-80c4-414f-8dc1-13b01d0e7c62-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:00.621841 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.621750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:00.621841 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.621788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdr4z\" (UniqueName: \"kubernetes.io/projected/0119b689-80c4-414f-8dc1-13b01d0e7c62-kube-api-access-xdr4z\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:00.621975 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:22:00.621877    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-serving-cert: secret "maas-api-e2e-ipp-b-8d29d7-serving-cert" not found
Sep 14 03:22:00.621975 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:22:00.621956    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-maas-api-tls podName:0119b689-80c4-414f-8dc1-13b01d0e7c62 nodeName:}" failed. No retries permitted until 2026-09-14 03:22:01.121936216 +0000 UTC m=+1787.847894259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-maas-api-tls") pod "maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f" (UID: "0119b689-80c4-414f-8dc1-13b01d0e7c62") : secret "maas-api-e2e-ipp-b-8d29d7-serving-cert" not found
Sep 14 03:22:00.622437 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.622411    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0119b689-80c4-414f-8dc1-13b01d0e7c62-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:00.624065 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.624019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-metrics-certs\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:00.631081 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:00.631056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdr4z\" (UniqueName: \"kubernetes.io/projected/0119b689-80c4-414f-8dc1-13b01d0e7c62-kube-api-access-xdr4z\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:01.124998 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:01.124964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:01.127298 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:01.127270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:01.369414 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:01.369380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:01.646261 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:01.646233    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"]
Sep 14 03:22:01.647203 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:22:01.647168    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0119b689_80c4_414f_8dc1_13b01d0e7c62.slice/crio-2af0df357729fcc187e609ec2913a04181901d49f02b0080243a0bea46e06160 WatchSource:0}: Error finding container 2af0df357729fcc187e609ec2913a04181901d49f02b0080243a0bea46e06160: Status 404 returned error can't find the container with id 2af0df357729fcc187e609ec2913a04181901d49f02b0080243a0bea46e06160
Sep 14 03:22:01.648755 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:01.648733    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 03:22:01.822577 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:01.822540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f" event={"ID":"0119b689-80c4-414f-8dc1-13b01d0e7c62","Type":"ContainerStarted","Data":"2af0df357729fcc187e609ec2913a04181901d49f02b0080243a0bea46e06160"}
Sep 14 03:22:02.830526 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:02.830494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f" event={"ID":"0119b689-80c4-414f-8dc1-13b01d0e7c62","Type":"ContainerStarted","Data":"d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb"}
Sep 14 03:22:02.830875 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:02.830553    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:02.909419 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:02.909369    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f" podStartSLOduration=2.5650005890000003 podStartE2EDuration="2.909353073s" totalImagesPullingTime="344.352484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:22:02.905731987 +0000 UTC m=+1789.631690037" watchObservedRunningTime="2026-09-14 03:22:02.909353073 +0000 UTC m=+1789.635311161"
Sep 14 03:22:04.967978 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:04.967943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:22:08.842189 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:08.842114    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:22:13.834160 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:13.834126    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 03:22:13.834556 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:13.834219    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 03:22:13.893322 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:13.893297    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:22:13.894320 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:13.894296    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:22:14.915792 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:14.915764    2580 generic.go:356] "Generic (PLEG): container finished" podID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerID="3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3" exitCode=0
Sep 14 03:22:14.916159 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:14.915836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" event={"ID":"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58","Type":"ContainerDied","Data":"3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3"}
Sep 14 03:22:14.916282 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:14.916268    2580 scope.go:117] "RemoveContainer" containerID="3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3"
Sep 14 03:22:14.917477 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:14.917444    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:22:14.917551 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:14.917537    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:22:14.917590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:14.917571    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 03:22:15.925496 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:15.925458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" event={"ID":"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58","Type":"ContainerStarted","Data":"455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c"}
Sep 14 03:22:16.886395 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:16.886356    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:22:16.886599 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:16.886573    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Sep 14 03:22:16.886644 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:16.886624    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Sep 14 03:22:17.885980 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:17.885944    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Sep 14 03:22:17.886389 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:17.885996    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Sep 14 03:22:18.885777 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:18.885741    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Sep 14 03:22:18.885950 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:18.885799    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Sep 14 03:22:19.886397 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:19.886333    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Sep 14 03:22:19.886819 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:19.886416    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Sep 14 03:22:20.886281 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:20.886241    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Sep 14 03:22:20.886488 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:20.886307    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Sep 14 03:22:21.997168 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:21.997138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:22:21.997544 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:21.997418    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:22:21.998116 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:21.998101    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:22:31.252639 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.252597    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"]
Sep 14 03:22:31.256540 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.256521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.258399 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.258379    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d081a3\"/\"openshift-service-ca.crt\""
Sep 14 03:22:31.258960 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.258945    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d081a3\"/\"default-dockercfg-2vb5b\""
Sep 14 03:22:31.259015 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.258969    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d081a3\"/\"kube-root-ca.crt\""
Sep 14 03:22:31.259089 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.259070    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d081a3\"/\"ipp-route-d081a3-kserve-self-signed-certs\""
Sep 14 03:22:31.270216 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.270187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"]
Sep 14 03:22:31.404341 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.404303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59ndx\" (UniqueName: \"kubernetes.io/projected/c801ce91-e707-4750-8397-3538906fd719-kube-api-access-59ndx\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.404341 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.404347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c801ce91-e707-4750-8397-3538906fd719-tls-certs\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.404553 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.404371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-home\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.404553 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.404504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-tmp-dir\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.404619 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.404553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-dshm\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.404654 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.404618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-model-cache\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.505851 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.505825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-model-cache\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.506017 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.505867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59ndx\" (UniqueName: \"kubernetes.io/projected/c801ce91-e707-4750-8397-3538906fd719-kube-api-access-59ndx\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.506017 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.505889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c801ce91-e707-4750-8397-3538906fd719-tls-certs\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.506136 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.506008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-home\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.506136 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.506122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-tmp-dir\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.506223 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.506171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-dshm\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.506329 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.506309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-model-cache\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.506376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.506355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-home\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.506445 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.506431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-tmp-dir\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.508319 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.508293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-dshm\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.508455 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.508349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c801ce91-e707-4750-8397-3538906fd719-tls-certs\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.515640 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.515614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59ndx\" (UniqueName: \"kubernetes.io/projected/c801ce91-e707-4750-8397-3538906fd719-kube-api-access-59ndx\") pod \"ipp-route-d081a3-kserve-664d8bcb87-46s6z\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") " pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.566960 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.566932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:31.841267 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:31.841244    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"]
Sep 14 03:22:31.842740 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:22:31.842712    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc801ce91_e707_4750_8397_3538906fd719.slice/crio-bd8fab36120ed489a6f1697ca32f7838f377f3be565244550eb5c6148ea2eced WatchSource:0}: Error finding container bd8fab36120ed489a6f1697ca32f7838f377f3be565244550eb5c6148ea2eced: Status 404 returned error can't find the container with id bd8fab36120ed489a6f1697ca32f7838f377f3be565244550eb5c6148ea2eced
Sep 14 03:22:32.020361 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:32.020325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z" event={"ID":"c801ce91-e707-4750-8397-3538906fd719","Type":"ContainerStarted","Data":"e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f"}
Sep 14 03:22:32.020522 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:32.020385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z" event={"ID":"c801ce91-e707-4750-8397-3538906fd719","Type":"ContainerStarted","Data":"bd8fab36120ed489a6f1697ca32f7838f377f3be565244550eb5c6148ea2eced"}
Sep 14 03:22:32.043643 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:32.043598    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z" podStartSLOduration=1.043582544 podStartE2EDuration="1.043582544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:22:32.04023003 +0000 UTC m=+1818.766188081" watchObservedRunningTime="2026-09-14 03:22:32.043582544 +0000 UTC m=+1818.769540595"
Sep 14 03:22:35.058863 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:35.058830    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:22:41.567568 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:41.567529    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:41.567568 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:41.567571    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:41.580178 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:41.580157    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:42.086612 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:42.086582    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:22:47.652681 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:47.652649    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:22:53.817096 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:22:53.817058    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l_f36f749f-26a5-4a01-9693-14b4a07e6906/payload-processing/0.log"
Sep 14 03:23:07.161792 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:07.161760    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:23:29.241877 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:29.241846    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"]
Sep 14 03:23:29.245405 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:29.245386    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"
Sep 14 03:23:29.247622 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:29.247603    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sv5sd\""
Sep 14 03:23:29.253937 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:29.253915    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"]
Sep 14 03:23:29.330662 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:29.330637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1768887d-b30f-4a6c-8341-a6a9a8daa099-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v\" (UID: \"1768887d-b30f-4a6c-8341-a6a9a8daa099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"
Sep 14 03:23:29.431836 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:29.431809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1768887d-b30f-4a6c-8341-a6a9a8daa099-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v\" (UID: \"1768887d-b30f-4a6c-8341-a6a9a8daa099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"
Sep 14 03:23:29.432424 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:29.432404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1768887d-b30f-4a6c-8341-a6a9a8daa099-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v\" (UID: \"1768887d-b30f-4a6c-8341-a6a9a8daa099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"
Sep 14 03:23:29.558172 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:29.558148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"
Sep 14 03:23:29.835242 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:29.835213    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"]
Sep 14 03:23:29.836517 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:23:29.836486    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1768887d_b30f_4a6c_8341_a6a9a8daa099.slice/crio-817f56f7a92fc68efa182dfa882a7a9b33659640f1cd7ef64a326168c83fa442 WatchSource:0}: Error finding container 817f56f7a92fc68efa182dfa882a7a9b33659640f1cd7ef64a326168c83fa442: Status 404 returned error can't find the container with id 817f56f7a92fc68efa182dfa882a7a9b33659640f1cd7ef64a326168c83fa442
Sep 14 03:23:30.349102 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:30.349069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v" event={"ID":"1768887d-b30f-4a6c-8341-a6a9a8daa099","Type":"ContainerStarted","Data":"590d6c2c5c3f09e684dd81119a1fb7e74a2c9692ed7b6c44b225c7deb2f4628e"}
Sep 14 03:23:30.349468 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:30.349125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v" event={"ID":"1768887d-b30f-4a6c-8341-a6a9a8daa099","Type":"ContainerStarted","Data":"817f56f7a92fc68efa182dfa882a7a9b33659640f1cd7ef64a326168c83fa442"}
Sep 14 03:23:30.364011 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:30.363917    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v" podStartSLOduration=1.363901818 podStartE2EDuration="1.363901818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:23:30.362890124 +0000 UTC m=+1877.088848176" watchObservedRunningTime="2026-09-14 03:23:30.363901818 +0000 UTC m=+1877.089859914"
Sep 14 03:23:31.356613 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:31.356581    2580 generic.go:356] "Generic (PLEG): container finished" podID="1768887d-b30f-4a6c-8341-a6a9a8daa099" containerID="590d6c2c5c3f09e684dd81119a1fb7e74a2c9692ed7b6c44b225c7deb2f4628e" exitCode=0
Sep 14 03:23:31.357099 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:31.356661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v" event={"ID":"1768887d-b30f-4a6c-8341-a6a9a8daa099","Type":"ContainerDied","Data":"590d6c2c5c3f09e684dd81119a1fb7e74a2c9692ed7b6c44b225c7deb2f4628e"}
Sep 14 03:23:32.497279 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:32.497258    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"
Sep 14 03:23:32.558055 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:32.558018    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1768887d-b30f-4a6c-8341-a6a9a8daa099-maas-api-service-ca\") pod \"1768887d-b30f-4a6c-8341-a6a9a8daa099\" (UID: \"1768887d-b30f-4a6c-8341-a6a9a8daa099\") "
Sep 14 03:23:32.558334 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:32.558315    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1768887d-b30f-4a6c-8341-a6a9a8daa099-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1768887d-b30f-4a6c-8341-a6a9a8daa099" (UID: "1768887d-b30f-4a6c-8341-a6a9a8daa099"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:23:32.658719 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:32.658660    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1768887d-b30f-4a6c-8341-a6a9a8daa099-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:23:33.368485 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:33.368450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v" event={"ID":"1768887d-b30f-4a6c-8341-a6a9a8daa099","Type":"ContainerDied","Data":"817f56f7a92fc68efa182dfa882a7a9b33659640f1cd7ef64a326168c83fa442"}
Sep 14 03:23:33.368485 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:33.368475    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"
Sep 14 03:23:33.368693 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:33.368520    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="817f56f7a92fc68efa182dfa882a7a9b33659640f1cd7ef64a326168c83fa442"
Sep 14 03:23:40.724417 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:40.724335    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"]
Sep 14 03:23:40.726746 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:40.726722    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b049b-mmb6v"]
Sep 14 03:23:41.756918 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:41.756887    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1768887d-b30f-4a6c-8341-a6a9a8daa099" path="/var/lib/kubelet/pods/1768887d-b30f-4a6c-8341-a6a9a8daa099/volumes"
Sep 14 03:23:45.057605 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:23:45.057573    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:24:04.662484 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:04.662446    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:24:07.149943 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.149910    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"]
Sep 14 03:24:07.150422 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.150172    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l" podUID="f36f749f-26a5-4a01-9693-14b4a07e6906" containerName="payload-processing" containerID="cri-o://ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad" gracePeriod=30
Sep 14 03:24:07.418956 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.418618    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:24:07.474356 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.474324    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36f749f-26a5-4a01-9693-14b4a07e6906-plugins-config-volume\") pod \"f36f749f-26a5-4a01-9693-14b4a07e6906\" (UID: \"f36f749f-26a5-4a01-9693-14b4a07e6906\") "
Sep 14 03:24:07.474506 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.474455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8jxj\" (UniqueName: \"kubernetes.io/projected/f36f749f-26a5-4a01-9693-14b4a07e6906-kube-api-access-z8jxj\") pod \"f36f749f-26a5-4a01-9693-14b4a07e6906\" (UID: \"f36f749f-26a5-4a01-9693-14b4a07e6906\") "
Sep 14 03:24:07.474752 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.474725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36f749f-26a5-4a01-9693-14b4a07e6906-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f36f749f-26a5-4a01-9693-14b4a07e6906" (UID: "f36f749f-26a5-4a01-9693-14b4a07e6906"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:24:07.476536 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.476511    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36f749f-26a5-4a01-9693-14b4a07e6906-kube-api-access-z8jxj" (OuterVolumeSpecName: "kube-api-access-z8jxj") pod "f36f749f-26a5-4a01-9693-14b4a07e6906" (UID: "f36f749f-26a5-4a01-9693-14b4a07e6906"). InnerVolumeSpecName "kube-api-access-z8jxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:24:07.568026 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.567992    2580 generic.go:356] "Generic (PLEG): container finished" podID="f36f749f-26a5-4a01-9693-14b4a07e6906" containerID="ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad" exitCode=0
Sep 14 03:24:07.568229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.568050    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"
Sep 14 03:24:07.568229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.568078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l" event={"ID":"f36f749f-26a5-4a01-9693-14b4a07e6906","Type":"ContainerDied","Data":"ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad"}
Sep 14 03:24:07.568229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.568141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l" event={"ID":"f36f749f-26a5-4a01-9693-14b4a07e6906","Type":"ContainerDied","Data":"0313c9240432e928122882b22ebaa232eb0a8f33e5ea8383fe0638332a2543ce"}
Sep 14 03:24:07.568229 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.568172    2580 scope.go:117] "RemoveContainer" containerID="ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad"
Sep 14 03:24:07.575799 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.575779    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36f749f-26a5-4a01-9693-14b4a07e6906-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:24:07.575799 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.575800    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8jxj\" (UniqueName: \"kubernetes.io/projected/f36f749f-26a5-4a01-9693-14b4a07e6906-kube-api-access-z8jxj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:24:07.577464 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.577448    2580 scope.go:117] "RemoveContainer" containerID="ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad"
Sep 14 03:24:07.577721 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:24:07.577704    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad\": container with ID starting with ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad not found: ID does not exist" containerID="ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad"
Sep 14 03:24:07.577766 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.577728    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad"} err="failed to get container status \"ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad\": rpc error: code = NotFound desc = could not find container \"ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad\": container with ID starting with ad8a108aed34cfa8494720b30ce4915ad802fc296a826d029ba73540e5ab42ad not found: ID does not exist"
Sep 14 03:24:07.593542 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.593518    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"]
Sep 14 03:24:07.597878 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.597853    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d081a3-5df4dd896c-dc97l"]
Sep 14 03:24:07.756834 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:07.756808    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f36f749f-26a5-4a01-9693-14b4a07e6906" path="/var/lib/kubelet/pods/f36f749f-26a5-4a01-9693-14b4a07e6906/volumes"
Sep 14 03:24:12.878869 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:12.878828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"]
Sep 14 03:24:17.674836 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:17.674802    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:24:27.897782 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:27.897745    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z" podUID="c801ce91-e707-4750-8397-3538906fd719" containerName="main" containerID="cri-o://e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f" gracePeriod=45
Sep 14 03:24:28.150436 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.150382    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:24:28.266307 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.266274    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c801ce91-e707-4750-8397-3538906fd719-tls-certs\") pod \"c801ce91-e707-4750-8397-3538906fd719\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") "
Sep 14 03:24:28.266460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.266340    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-home\") pod \"c801ce91-e707-4750-8397-3538906fd719\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") "
Sep 14 03:24:28.266460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.266360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-model-cache\") pod \"c801ce91-e707-4750-8397-3538906fd719\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") "
Sep 14 03:24:28.266460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.266377    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-tmp-dir\") pod \"c801ce91-e707-4750-8397-3538906fd719\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") "
Sep 14 03:24:28.266460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.266442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-dshm\") pod \"c801ce91-e707-4750-8397-3538906fd719\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") "
Sep 14 03:24:28.266631 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.266462    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59ndx\" (UniqueName: \"kubernetes.io/projected/c801ce91-e707-4750-8397-3538906fd719-kube-api-access-59ndx\") pod \"c801ce91-e707-4750-8397-3538906fd719\" (UID: \"c801ce91-e707-4750-8397-3538906fd719\") "
Sep 14 03:24:28.266631 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.266595    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-home" (OuterVolumeSpecName: "home") pod "c801ce91-e707-4750-8397-3538906fd719" (UID: "c801ce91-e707-4750-8397-3538906fd719"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:24:28.266740 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.266651    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-model-cache" (OuterVolumeSpecName: "model-cache") pod "c801ce91-e707-4750-8397-3538906fd719" (UID: "c801ce91-e707-4750-8397-3538906fd719"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:24:28.266805 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.266757    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c801ce91-e707-4750-8397-3538906fd719" (UID: "c801ce91-e707-4750-8397-3538906fd719"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:24:28.266869 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.266808    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:24:28.266869 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.266826    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:24:28.268498 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.268476    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c801ce91-e707-4750-8397-3538906fd719-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c801ce91-e707-4750-8397-3538906fd719" (UID: "c801ce91-e707-4750-8397-3538906fd719"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:24:28.268575 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.268536    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c801ce91-e707-4750-8397-3538906fd719-kube-api-access-59ndx" (OuterVolumeSpecName: "kube-api-access-59ndx") pod "c801ce91-e707-4750-8397-3538906fd719" (UID: "c801ce91-e707-4750-8397-3538906fd719"). InnerVolumeSpecName "kube-api-access-59ndx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:24:28.268626 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.268604    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-dshm" (OuterVolumeSpecName: "dshm") pod "c801ce91-e707-4750-8397-3538906fd719" (UID: "c801ce91-e707-4750-8397-3538906fd719"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:24:28.367376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.367344    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:24:28.367376 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.367369    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c801ce91-e707-4750-8397-3538906fd719-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:24:28.367545 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.367381    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59ndx\" (UniqueName: \"kubernetes.io/projected/c801ce91-e707-4750-8397-3538906fd719-kube-api-access-59ndx\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:24:28.367545 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.367394    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c801ce91-e707-4750-8397-3538906fd719-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:24:28.687308 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.687266    2580 generic.go:356] "Generic (PLEG): container finished" podID="c801ce91-e707-4750-8397-3538906fd719" containerID="e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f" exitCode=0
Sep 14 03:24:28.687308 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.687307    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"
Sep 14 03:24:28.687539 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.687318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z" event={"ID":"c801ce91-e707-4750-8397-3538906fd719","Type":"ContainerDied","Data":"e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f"}
Sep 14 03:24:28.687539 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.687356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z" event={"ID":"c801ce91-e707-4750-8397-3538906fd719","Type":"ContainerDied","Data":"bd8fab36120ed489a6f1697ca32f7838f377f3be565244550eb5c6148ea2eced"}
Sep 14 03:24:28.687539 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.687374    2580 scope.go:117] "RemoveContainer" containerID="e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f"
Sep 14 03:24:28.696418 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.696396    2580 scope.go:117] "RemoveContainer" containerID="e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f"
Sep 14 03:24:28.696723 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:24:28.696700    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f\": container with ID starting with e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f not found: ID does not exist" containerID="e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f"
Sep 14 03:24:28.696851 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.696731    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f"} err="failed to get container status \"e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f\": rpc error: code = NotFound desc = could not find container \"e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f\": container with ID starting with e5dec26163bcd333628b132ce2c64a77ac4035a013c77f7c97bf3a91e566a99f not found: ID does not exist"
Sep 14 03:24:28.710100 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.710078    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"]
Sep 14 03:24:28.715951 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:28.715929    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d081a3/ipp-route-d081a3-kserve-664d8bcb87-46s6z"]
Sep 14 03:24:29.756258 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:29.756222    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c801ce91-e707-4750-8397-3538906fd719" path="/var/lib/kubelet/pods/c801ce91-e707-4750-8397-3538906fd719/volumes"
Sep 14 03:24:56.253051 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:56.252999    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"]
Sep 14 03:24:56.253570 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:24:56.253272    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy" containerID="cri-o://e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619" gracePeriod=30
Sep 14 03:25:01.502852 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.502827    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:25:01.551559 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.551498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-credential-socket\") pod \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") "
Sep 14 03:25:01.551678 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.551561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-certs\") pod \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") "
Sep 14 03:25:01.551678 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.551606    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-envoy\") pod \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") "
Sep 14 03:25:01.551678 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.551630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-data\") pod \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") "
Sep 14 03:25:01.551678 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.551651    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-podinfo\") pod \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") "
Sep 14 03:25:01.551678 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.551666    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istiod-ca-cert\") pod \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") "
Sep 14 03:25:01.551957 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.551759    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b0f64539-7c06-43b8-9b0a-d875804e9e9b" (UID: "b0f64539-7c06-43b8-9b0a-d875804e9e9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:25:01.551957 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.551836    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b0f64539-7c06-43b8-9b0a-d875804e9e9b" (UID: "b0f64539-7c06-43b8-9b0a-d875804e9e9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:25:01.551957 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.551924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-token\") pod \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") "
Sep 14 03:25:01.552147 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.551960    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt424\" (UniqueName: \"kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-kube-api-access-wt424\") pod \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") "
Sep 14 03:25:01.552147 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.551972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b0f64539-7c06-43b8-9b0a-d875804e9e9b" (UID: "b0f64539-7c06-43b8-9b0a-d875804e9e9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:25:01.552147 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.552008    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-socket\") pod \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\" (UID: \"b0f64539-7c06-43b8-9b0a-d875804e9e9b\") "
Sep 14 03:25:01.552147 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.552064    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b0f64539-7c06-43b8-9b0a-d875804e9e9b" (UID: "b0f64539-7c06-43b8-9b0a-d875804e9e9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:25:01.552380 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.552320    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:01.552380 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.552340    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:01.552380 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.552354    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:01.552380 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.552368    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:01.552595 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.552403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b0f64539-7c06-43b8-9b0a-d875804e9e9b" (UID: "b0f64539-7c06-43b8-9b0a-d875804e9e9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:25:01.553879 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.553844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b0f64539-7c06-43b8-9b0a-d875804e9e9b" (UID: "b0f64539-7c06-43b8-9b0a-d875804e9e9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 03:25:01.553999 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.553975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-kube-api-access-wt424" (OuterVolumeSpecName: "kube-api-access-wt424") pod "b0f64539-7c06-43b8-9b0a-d875804e9e9b" (UID: "b0f64539-7c06-43b8-9b0a-d875804e9e9b"). InnerVolumeSpecName "kube-api-access-wt424". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:01.554208 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.554180    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b0f64539-7c06-43b8-9b0a-d875804e9e9b" (UID: "b0f64539-7c06-43b8-9b0a-d875804e9e9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:25:01.554304 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.554237    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b0f64539-7c06-43b8-9b0a-d875804e9e9b" (UID: "b0f64539-7c06-43b8-9b0a-d875804e9e9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:01.653192 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.653166    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:01.653192 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.653189    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:01.653324 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.653199    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:01.653324 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.653207    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:01.653324 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.653214    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt424\" (UniqueName: \"kubernetes.io/projected/b0f64539-7c06-43b8-9b0a-d875804e9e9b-kube-api-access-wt424\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:01.864575 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.864489    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerID="e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619" exitCode=0
Sep 14 03:25:01.864575 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.864536    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"
Sep 14 03:25:01.864784 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.864568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" event={"ID":"b0f64539-7c06-43b8-9b0a-d875804e9e9b","Type":"ContainerDied","Data":"e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619"}
Sep 14 03:25:01.864784 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.864619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh" event={"ID":"b0f64539-7c06-43b8-9b0a-d875804e9e9b","Type":"ContainerDied","Data":"27bae8c8fc7924a91550486171e37558bc4de95a00cd14402dc8b82f7eaefebd"}
Sep 14 03:25:01.864784 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.864640    2580 scope.go:117] "RemoveContainer" containerID="e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619"
Sep 14 03:25:01.876143 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.876122    2580 scope.go:117] "RemoveContainer" containerID="600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967"
Sep 14 03:25:01.885805 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.885770    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"]
Sep 14 03:25:01.885873 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.885860    2580 scope.go:117] "RemoveContainer" containerID="e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619"
Sep 14 03:25:01.886232 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:25:01.886206    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619\": container with ID starting with e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619 not found: ID does not exist" containerID="e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619"
Sep 14 03:25:01.886350 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.886240    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619"} err="failed to get container status \"e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619\": rpc error: code = NotFound desc = could not find container \"e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619\": container with ID starting with e06d2a4d1eaf8cd43f74807b261c6dbc9dd08abd7903140d8e7df2abebe4c619 not found: ID does not exist"
Sep 14 03:25:01.886350 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.886262    2580 scope.go:117] "RemoveContainer" containerID="600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967"
Sep 14 03:25:01.886628 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:25:01.886591    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967\": container with ID starting with 600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967 not found: ID does not exist" containerID="600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967"
Sep 14 03:25:01.886628 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.886632    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967"} err="failed to get container status \"600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967\": rpc error: code = NotFound desc = could not find container \"600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967\": container with ID starting with 600a8a1d43042782d20fcc47930bf8ba1373d1b5e283f02f64f8fdf6f3567967 not found: ID does not exist"
Sep 14 03:25:01.889150 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:01.889131    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d081a3-openshift-default-7d588845bd-fjfxh"]
Sep 14 03:25:03.763250 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:03.763215    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" path="/var/lib/kubelet/pods/b0f64539-7c06-43b8-9b0a-d875804e9e9b/volumes"
Sep 14 03:25:07.154232 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.154147    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"]
Sep 14 03:25:07.154704 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.154387    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f" podUID="0119b689-80c4-414f-8dc1-13b01d0e7c62" containerName="maas-api" containerID="cri-o://d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb" gracePeriod=30
Sep 14 03:25:07.399109 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.399084    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:25:07.503099 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.503002    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-maas-api-tls\") pod \"0119b689-80c4-414f-8dc1-13b01d0e7c62\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") "
Sep 14 03:25:07.503099 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.503097    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0119b689-80c4-414f-8dc1-13b01d0e7c62-openshift-service-ca\") pod \"0119b689-80c4-414f-8dc1-13b01d0e7c62\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") "
Sep 14 03:25:07.503297 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.503143    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-metrics-certs\") pod \"0119b689-80c4-414f-8dc1-13b01d0e7c62\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") "
Sep 14 03:25:07.503297 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.503175    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdr4z\" (UniqueName: \"kubernetes.io/projected/0119b689-80c4-414f-8dc1-13b01d0e7c62-kube-api-access-xdr4z\") pod \"0119b689-80c4-414f-8dc1-13b01d0e7c62\" (UID: \"0119b689-80c4-414f-8dc1-13b01d0e7c62\") "
Sep 14 03:25:07.503482 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.503457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0119b689-80c4-414f-8dc1-13b01d0e7c62-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0119b689-80c4-414f-8dc1-13b01d0e7c62" (UID: "0119b689-80c4-414f-8dc1-13b01d0e7c62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:25:07.505343 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.505308    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0119b689-80c4-414f-8dc1-13b01d0e7c62" (UID: "0119b689-80c4-414f-8dc1-13b01d0e7c62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:25:07.505455 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.505339    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0119b689-80c4-414f-8dc1-13b01d0e7c62" (UID: "0119b689-80c4-414f-8dc1-13b01d0e7c62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:25:07.505455 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.505387    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0119b689-80c4-414f-8dc1-13b01d0e7c62-kube-api-access-xdr4z" (OuterVolumeSpecName: "kube-api-access-xdr4z") pod "0119b689-80c4-414f-8dc1-13b01d0e7c62" (UID: "0119b689-80c4-414f-8dc1-13b01d0e7c62"). InnerVolumeSpecName "kube-api-access-xdr4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:07.603954 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.603913    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:07.603954 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.603950    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdr4z\" (UniqueName: \"kubernetes.io/projected/0119b689-80c4-414f-8dc1-13b01d0e7c62-kube-api-access-xdr4z\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:07.604202 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.603965    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0119b689-80c4-414f-8dc1-13b01d0e7c62-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:07.604202 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.603980    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0119b689-80c4-414f-8dc1-13b01d0e7c62-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:07.898494 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.898459    2580 generic.go:356] "Generic (PLEG): container finished" podID="0119b689-80c4-414f-8dc1-13b01d0e7c62" containerID="d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb" exitCode=0
Sep 14 03:25:07.898667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.898501    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"
Sep 14 03:25:07.898667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.898542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f" event={"ID":"0119b689-80c4-414f-8dc1-13b01d0e7c62","Type":"ContainerDied","Data":"d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb"}
Sep 14 03:25:07.898667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.898595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f" event={"ID":"0119b689-80c4-414f-8dc1-13b01d0e7c62","Type":"ContainerDied","Data":"2af0df357729fcc187e609ec2913a04181901d49f02b0080243a0bea46e06160"}
Sep 14 03:25:07.898667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.898622    2580 scope.go:117] "RemoveContainer" containerID="d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb"
Sep 14 03:25:07.908085 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.908059    2580 scope.go:117] "RemoveContainer" containerID="d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb"
Sep 14 03:25:07.908360 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:25:07.908341    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb\": container with ID starting with d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb not found: ID does not exist" containerID="d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb"
Sep 14 03:25:07.908428 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.908367    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb"} err="failed to get container status \"d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb\": rpc error: code = NotFound desc = could not find container \"d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb\": container with ID starting with d710d6288f2b8745514512c04aeffcb928bc30c9f91f56f87184c351bc6e4afb not found: ID does not exist"
Sep 14 03:25:07.917858 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.917837    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"]
Sep 14 03:25:07.922563 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:07.922545    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d29d7-545c5487fd-5sx6f"]
Sep 14 03:25:09.756978 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:09.756942    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0119b689-80c4-414f-8dc1-13b01d0e7c62" path="/var/lib/kubelet/pods/0119b689-80c4-414f-8dc1-13b01d0e7c62/volumes"
Sep 14 03:25:10.158125 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:10.158046    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:25:13.940678 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:13.940643    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"]
Sep 14 03:25:13.941091 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:13.940895    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy" containerID="cri-o://455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c" gracePeriod=30
Sep 14 03:25:14.325556 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325524    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"]
Sep 14 03:25:14.325870 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325858    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0119b689-80c4-414f-8dc1-13b01d0e7c62" containerName="maas-api"
Sep 14 03:25:14.325916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325872    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0119b689-80c4-414f-8dc1-13b01d0e7c62" containerName="maas-api"
Sep 14 03:25:14.325916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325883    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy"
Sep 14 03:25:14.325916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325889    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy"
Sep 14 03:25:14.325916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325897    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy"
Sep 14 03:25:14.325916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325902    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy"
Sep 14 03:25:14.325916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325912    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c801ce91-e707-4750-8397-3538906fd719" containerName="main"
Sep 14 03:25:14.325916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325917    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c801ce91-e707-4750-8397-3538906fd719" containerName="main"
Sep 14 03:25:14.326162 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325928    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36f749f-26a5-4a01-9693-14b4a07e6906" containerName="payload-processing"
Sep 14 03:25:14.326162 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325933    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36f749f-26a5-4a01-9693-14b4a07e6906" containerName="payload-processing"
Sep 14 03:25:14.326162 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325941    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1768887d-b30f-4a6c-8341-a6a9a8daa099" containerName="revoke-keys"
Sep 14 03:25:14.326162 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.325946    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1768887d-b30f-4a6c-8341-a6a9a8daa099" containerName="revoke-keys"
Sep 14 03:25:14.326162 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.326000    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36f749f-26a5-4a01-9693-14b4a07e6906" containerName="payload-processing"
Sep 14 03:25:14.326162 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.326007    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy"
Sep 14 03:25:14.326162 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.326013    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c801ce91-e707-4750-8397-3538906fd719" containerName="main"
Sep 14 03:25:14.326162 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.326020    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f64539-7c06-43b8-9b0a-d875804e9e9b" containerName="istio-proxy"
Sep 14 03:25:14.326162 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.326025    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0119b689-80c4-414f-8dc1-13b01d0e7c62" containerName="maas-api"
Sep 14 03:25:14.326162 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.326044    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1768887d-b30f-4a6c-8341-a6a9a8daa099" containerName="revoke-keys"
Sep 14 03:25:14.330316 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.330301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"
Sep 14 03:25:14.334183 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.334158    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sv5sd\""
Sep 14 03:25:14.334413 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.334398    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:25:14.344326 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.344306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"]
Sep 14 03:25:14.364570 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.364545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a96dfb2-9c37-4282-ba39-ccbd4465fc6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r\" (UID: \"8a96dfb2-9c37-4282-ba39-ccbd4465fc6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"
Sep 14 03:25:14.465684 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.465613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a96dfb2-9c37-4282-ba39-ccbd4465fc6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r\" (UID: \"8a96dfb2-9c37-4282-ba39-ccbd4465fc6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"
Sep 14 03:25:14.466167 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.466151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a96dfb2-9c37-4282-ba39-ccbd4465fc6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r\" (UID: \"8a96dfb2-9c37-4282-ba39-ccbd4465fc6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"
Sep 14 03:25:14.640125 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.640096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"
Sep 14 03:25:14.909359 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.909331    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"]
Sep 14 03:25:14.910376 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:25:14.910351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a96dfb2_9c37_4282_ba39_ccbd4465fc6f.slice/crio-334ca50c09e69ede515d6acd512428f30e81b85f0cbf2d53a5b858b0570678e4 WatchSource:0}: Error finding container 334ca50c09e69ede515d6acd512428f30e81b85f0cbf2d53a5b858b0570678e4: Status 404 returned error can't find the container with id 334ca50c09e69ede515d6acd512428f30e81b85f0cbf2d53a5b858b0570678e4
Sep 14 03:25:14.945590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:14.945568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r" event={"ID":"8a96dfb2-9c37-4282-ba39-ccbd4465fc6f","Type":"ContainerStarted","Data":"334ca50c09e69ede515d6acd512428f30e81b85f0cbf2d53a5b858b0570678e4"}
Sep 14 03:25:15.952475 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:15.952384    2580 generic.go:356] "Generic (PLEG): container finished" podID="8a96dfb2-9c37-4282-ba39-ccbd4465fc6f" containerID="d7af5bf23d488939b2521ac75c3b87dd704defceb604bc52773320cec8e99be0" exitCode=0
Sep 14 03:25:15.952475 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:15.952440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r" event={"ID":"8a96dfb2-9c37-4282-ba39-ccbd4465fc6f","Type":"ContainerDied","Data":"d7af5bf23d488939b2521ac75c3b87dd704defceb604bc52773320cec8e99be0"}
Sep 14 03:25:17.091629 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:17.091605    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"
Sep 14 03:25:17.191361 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:17.191330    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a96dfb2-9c37-4282-ba39-ccbd4465fc6f-maas-api-service-ca\") pod \"8a96dfb2-9c37-4282-ba39-ccbd4465fc6f\" (UID: \"8a96dfb2-9c37-4282-ba39-ccbd4465fc6f\") "
Sep 14 03:25:17.191663 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:17.191641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a96dfb2-9c37-4282-ba39-ccbd4465fc6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a96dfb2-9c37-4282-ba39-ccbd4465fc6f" (UID: "8a96dfb2-9c37-4282-ba39-ccbd4465fc6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:25:17.292706 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:17.292677    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a96dfb2-9c37-4282-ba39-ccbd4465fc6f-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:17.963995 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:17.963964    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"
Sep 14 03:25:17.964166 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:17.963967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r" event={"ID":"8a96dfb2-9c37-4282-ba39-ccbd4465fc6f","Type":"ContainerDied","Data":"334ca50c09e69ede515d6acd512428f30e81b85f0cbf2d53a5b858b0570678e4"}
Sep 14 03:25:17.964166 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:17.964094    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="334ca50c09e69ede515d6acd512428f30e81b85f0cbf2d53a5b858b0570678e4"
Sep 14 03:25:19.189221 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.189199    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:25:19.309171 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309145    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdpf5\" (UniqueName: \"kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-kube-api-access-vdpf5\") pod \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") "
Sep 14 03:25:19.309314 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309201    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-credential-socket\") pod \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") "
Sep 14 03:25:19.309314 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-token\") pod \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") "
Sep 14 03:25:19.309314 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-podinfo\") pod \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") "
Sep 14 03:25:19.309460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309347    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-envoy\") pod \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") "
Sep 14 03:25:19.309460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309392    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-certs\") pod \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") "
Sep 14 03:25:19.309460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309420    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-data\") pod \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") "
Sep 14 03:25:19.309460 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istiod-ca-cert\") pod \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") "
Sep 14 03:25:19.309674 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309466    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-socket\") pod \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\" (UID: \"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58\") "
Sep 14 03:25:19.309674 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309497    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" (UID: "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:25:19.309788 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309710    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:19.309788 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309712    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" (UID: "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:25:19.309788 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309721    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" (UID: "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:25:19.309955 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.309915    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" (UID: "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:25:19.310081 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.310020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" (UID: "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:25:19.311498 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.311469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-kube-api-access-vdpf5" (OuterVolumeSpecName: "kube-api-access-vdpf5") pod "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" (UID: "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58"). InnerVolumeSpecName "kube-api-access-vdpf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:19.311609 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.311496    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" (UID: "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 03:25:19.311609 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.311529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" (UID: "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:19.311733 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.311716    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" (UID: "9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:25:19.411077 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.411055    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:19.411077 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.411075    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:19.411208 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.411084    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:19.411208 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.411091    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:19.411208 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.411098    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:19.411208 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.411106    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:19.411208 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.411117    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:19.411208 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.411127    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdpf5\" (UniqueName: \"kubernetes.io/projected/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58-kube-api-access-vdpf5\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:19.976482 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.976445    2580 generic.go:356] "Generic (PLEG): container finished" podID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerID="455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c" exitCode=0
Sep 14 03:25:19.976676 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.976491    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"
Sep 14 03:25:19.976676 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.976533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" event={"ID":"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58","Type":"ContainerDied","Data":"455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c"}
Sep 14 03:25:19.976676 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.976592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg" event={"ID":"9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58","Type":"ContainerDied","Data":"4ec70f6168b1474724291d661b81cd03847a8ef025c9ffbcb939fc6c5878635a"}
Sep 14 03:25:19.976676 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.976624    2580 scope.go:117] "RemoveContainer" containerID="455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c"
Sep 14 03:25:19.985794 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.985776    2580 scope.go:117] "RemoveContainer" containerID="3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3"
Sep 14 03:25:19.994414 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.994393    2580 scope.go:117] "RemoveContainer" containerID="455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c"
Sep 14 03:25:19.994665 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:25:19.994649    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c\": container with ID starting with 455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c not found: ID does not exist" containerID="455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c"
Sep 14 03:25:19.994751 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.994672    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c"} err="failed to get container status \"455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c\": rpc error: code = NotFound desc = could not find container \"455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c\": container with ID starting with 455de1b2bde559aa1fda0ecc3315f73a24b52164d8f23ca346729c2604831c8c not found: ID does not exist"
Sep 14 03:25:19.994751 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.994687    2580 scope.go:117] "RemoveContainer" containerID="3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3"
Sep 14 03:25:19.994964 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:25:19.994945    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3\": container with ID starting with 3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3 not found: ID does not exist" containerID="3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3"
Sep 14 03:25:19.995008 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.994968    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3"} err="failed to get container status \"3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3\": rpc error: code = NotFound desc = could not find container \"3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3\": container with ID starting with 3c426f442d90a53996514fffab53249c6fb08351296cd28afa39a63990e153b3 not found: ID does not exist"
Sep 14 03:25:19.996471 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:19.996451    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"]
Sep 14 03:25:20.001206 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:20.001182    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d29d7-openshift-default-556cb546d4-dzwhg"]
Sep 14 03:25:21.756485 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:21.756450    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" path="/var/lib/kubelet/pods/9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58/volumes"
Sep 14 03:25:25.014451 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:25.014414    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg"]
Sep 14 03:25:25.018331 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:25.018303    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b32aa-29822595-mwlgg"]
Sep 14 03:25:25.285544 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:25.285456    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"]
Sep 14 03:25:25.287596 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:25.287575    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b32aa-tjf9r"]
Sep 14 03:25:25.758425 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:25.758390    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ff5e95-0418-4412-8c1d-dd5f6a6bb5bb" path="/var/lib/kubelet/pods/24ff5e95-0418-4412-8c1d-dd5f6a6bb5bb/volumes"
Sep 14 03:25:25.758867 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:25.758833    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a96dfb2-9c37-4282-ba39-ccbd4465fc6f" path="/var/lib/kubelet/pods/8a96dfb2-9c37-4282-ba39-ccbd4465fc6f/volumes"
Sep 14 03:25:28.097717 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:28.097681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:25:44.023667 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:44.023635    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"]
Sep 14 03:25:44.024214 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:44.023825    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9" podUID="9a957b12-750d-470c-8516-9b613f12f011" containerName="payload-processing" containerID="cri-o://a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0" gracePeriod=30
Sep 14 03:25:44.270213 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:44.270189    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:25:44.409641 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:44.409567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a957b12-750d-470c-8516-9b613f12f011-plugins-config-volume\") pod \"9a957b12-750d-470c-8516-9b613f12f011\" (UID: \"9a957b12-750d-470c-8516-9b613f12f011\") "
Sep 14 03:25:44.409788 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:44.409647    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcr4h\" (UniqueName: \"kubernetes.io/projected/9a957b12-750d-470c-8516-9b613f12f011-kube-api-access-bcr4h\") pod \"9a957b12-750d-470c-8516-9b613f12f011\" (UID: \"9a957b12-750d-470c-8516-9b613f12f011\") "
Sep 14 03:25:44.409948 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:44.409925    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a957b12-750d-470c-8516-9b613f12f011-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a957b12-750d-470c-8516-9b613f12f011" (UID: "9a957b12-750d-470c-8516-9b613f12f011"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:25:44.411728 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:44.411699    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a957b12-750d-470c-8516-9b613f12f011-kube-api-access-bcr4h" (OuterVolumeSpecName: "kube-api-access-bcr4h") pod "9a957b12-750d-470c-8516-9b613f12f011" (UID: "9a957b12-750d-470c-8516-9b613f12f011"). InnerVolumeSpecName "kube-api-access-bcr4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:44.510658 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:44.510621    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a957b12-750d-470c-8516-9b613f12f011-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:44.510658 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:44.510650    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcr4h\" (UniqueName: \"kubernetes.io/projected/9a957b12-750d-470c-8516-9b613f12f011-kube-api-access-bcr4h\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:25:45.110356 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:45.110319    2580 generic.go:356] "Generic (PLEG): container finished" podID="9a957b12-750d-470c-8516-9b613f12f011" containerID="a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0" exitCode=0
Sep 14 03:25:45.110730 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:45.110361    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"
Sep 14 03:25:45.110730 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:45.110403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9" event={"ID":"9a957b12-750d-470c-8516-9b613f12f011","Type":"ContainerDied","Data":"a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0"}
Sep 14 03:25:45.110730 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:45.110453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9" event={"ID":"9a957b12-750d-470c-8516-9b613f12f011","Type":"ContainerDied","Data":"2b56efedbd80eb728a3c7f9a0a658569c10b13eec7e5280d89adb61613d97ca1"}
Sep 14 03:25:45.110730 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:45.110483    2580 scope.go:117] "RemoveContainer" containerID="a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0"
Sep 14 03:25:45.119750 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:45.119731    2580 scope.go:117] "RemoveContainer" containerID="a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0"
Sep 14 03:25:45.120021 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:25:45.120002    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0\": container with ID starting with a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0 not found: ID does not exist" containerID="a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0"
Sep 14 03:25:45.120090 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:45.120044    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0"} err="failed to get container status \"a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0\": rpc error: code = NotFound desc = could not find container \"a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0\": container with ID starting with a16eb5b54004fe9d0c079fc5d350955cad14ad311e58e74d8384f496d17424c0 not found: ID does not exist"
Sep 14 03:25:45.133201 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:45.133181    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"]
Sep 14 03:25:45.144146 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:45.144121    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-177593-6c4bf64dcc-2t8c9"]
Sep 14 03:25:45.756578 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:45.756544    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a957b12-750d-470c-8516-9b613f12f011" path="/var/lib/kubelet/pods/9a957b12-750d-470c-8516-9b613f12f011/volumes"
Sep 14 03:25:46.366342 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:46.366308    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:25:55.656912 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:25:55.656880    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:26:14.203366 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:26:14.203335    2580 scope.go:117] "RemoveContainer" containerID="fb27247e91617c22415fa85d7c3059950f17730354d1b47f4a2a6e379f2add1d"
Sep 14 03:27:13.937975 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:27:13.937950    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:27:13.938466 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:27:13.937960    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:27:16.251415 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:27:16.251378    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:27:43.056311 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:27:43.056278    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:27:50.956149 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:27:50.956112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:27:58.158920 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:27:58.158882    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:28:05.250218 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:28:05.250181    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:28:20.153416 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:28:20.153334    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:28:29.851638 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:28:29.851609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:29:54.247747 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:29:54.247712    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:30:00.951446 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:00.951410    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:30:08.956156 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:08.956123    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:30:14.256362 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:14.256332    2580 scope.go:117] "RemoveContainer" containerID="590d6c2c5c3f09e684dd81119a1fb7e74a2c9692ed7b6c44b225c7deb2f4628e"
Sep 14 03:30:17.671964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.671936    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"]
Sep 14 03:30:17.672365 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672282    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy"
Sep 14 03:30:17.672365 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672294    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy"
Sep 14 03:30:17.672365 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672306    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy"
Sep 14 03:30:17.672365 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672312    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy"
Sep 14 03:30:17.672365 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672321    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a96dfb2-9c37-4282-ba39-ccbd4465fc6f" containerName="revoke-keys"
Sep 14 03:30:17.672365 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672327    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a96dfb2-9c37-4282-ba39-ccbd4465fc6f" containerName="revoke-keys"
Sep 14 03:30:17.672365 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672334    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a957b12-750d-470c-8516-9b613f12f011" containerName="payload-processing"
Sep 14 03:30:17.672365 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672339    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a957b12-750d-470c-8516-9b613f12f011" containerName="payload-processing"
Sep 14 03:30:17.672658 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672398    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a957b12-750d-470c-8516-9b613f12f011" containerName="payload-processing"
Sep 14 03:30:17.672658 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672407    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy"
Sep 14 03:30:17.672658 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672413    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c6a250f-ebe5-4a4c-93c5-6ab9a8d16f58" containerName="istio-proxy"
Sep 14 03:30:17.672658 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.672420    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a96dfb2-9c37-4282-ba39-ccbd4465fc6f" containerName="revoke-keys"
Sep 14 03:30:17.676601 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.676585    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.679141 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.679121    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-67807548-kserve-self-signed-certs\""
Sep 14 03:30:17.684918 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.684895    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"]
Sep 14 03:30:17.699752 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.699726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-dshm\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.699870 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.699772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7dv\" (UniqueName: \"kubernetes.io/projected/b18a09b6-e52c-4195-a06d-1537c3ded8bc-kube-api-access-xn7dv\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.699921 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.699867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-home\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.699987 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.699929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-model-cache\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.699987 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.699961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tmp-dir\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.700104 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.700021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tls-certs\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.800507 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.800476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-home\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.800660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.800520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-model-cache\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.800660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.800536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tmp-dir\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.800660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.800580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tls-certs\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.800660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.800603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-dshm\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.800660 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.800632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7dv\" (UniqueName: \"kubernetes.io/projected/b18a09b6-e52c-4195-a06d-1537c3ded8bc-kube-api-access-xn7dv\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.800926 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.800891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-model-cache\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.801003 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.800948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-home\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.801076 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.800987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tmp-dir\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.802937 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.802919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-dshm\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.803248 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.803226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tls-certs\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.816000 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.815969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7dv\" (UniqueName: \"kubernetes.io/projected/b18a09b6-e52c-4195-a06d-1537c3ded8bc-kube-api-access-xn7dv\") pod \"e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") " pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:17.986825 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:17.986750    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:18.262334 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:18.262147    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"]
Sep 14 03:30:18.264893 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:30:18.264863    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb18a09b6_e52c_4195_a06d_1537c3ded8bc.slice/crio-fee77e2fda013640efe2799a5e5901d1ceb2ca5dd859ad88b78bc21c9d30ea9a WatchSource:0}: Error finding container fee77e2fda013640efe2799a5e5901d1ceb2ca5dd859ad88b78bc21c9d30ea9a: Status 404 returned error can't find the container with id fee77e2fda013640efe2799a5e5901d1ceb2ca5dd859ad88b78bc21c9d30ea9a
Sep 14 03:30:18.548645 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:18.548569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q" event={"ID":"b18a09b6-e52c-4195-a06d-1537c3ded8bc","Type":"ContainerStarted","Data":"145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a"}
Sep 14 03:30:18.548645 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:18.548614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q" event={"ID":"b18a09b6-e52c-4195-a06d-1537c3ded8bc","Type":"ContainerStarted","Data":"fee77e2fda013640efe2799a5e5901d1ceb2ca5dd859ad88b78bc21c9d30ea9a"}
Sep 14 03:30:18.570853 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:18.570790    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q" podStartSLOduration=1.570766012 podStartE2EDuration="1.570766012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:30:18.565438138 +0000 UTC m=+2285.291396194" watchObservedRunningTime="2026-09-14 03:30:18.570766012 +0000 UTC m=+2285.296724065"
Sep 14 03:30:20.156309 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:20.156277    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:30:22.047023 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.046995    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"]
Sep 14 03:30:22.050328 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.050312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.052341 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.052320    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-67807548-kserve-self-signed-certs\""
Sep 14 03:30:22.059118 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.059098    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"]
Sep 14 03:30:22.131734 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.131701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-home\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.131885 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.131753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-tmp-dir\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.131885 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.131822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjdwm\" (UniqueName: \"kubernetes.io/projected/7596f7ca-55e6-4a9f-9b64-8e887451da62-kube-api-access-sjdwm\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.131885 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.131853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-dshm\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.131885 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.131878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7596f7ca-55e6-4a9f-9b64-8e887451da62-tls-certs\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.132085 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.131986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-model-cache\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.233080 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.233024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjdwm\" (UniqueName: \"kubernetes.io/projected/7596f7ca-55e6-4a9f-9b64-8e887451da62-kube-api-access-sjdwm\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.233242 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.233091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-dshm\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.233242 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.233119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7596f7ca-55e6-4a9f-9b64-8e887451da62-tls-certs\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.233242 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.233183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-model-cache\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.233368 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.233253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-home\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.233368 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.233284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-tmp-dir\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.233619 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.233587    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-home\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.233737 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.233622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-model-cache\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.233737 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.233653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-tmp-dir\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.235381 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.235361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-dshm\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.235624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.235607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7596f7ca-55e6-4a9f-9b64-8e887451da62-tls-certs\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.239950 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.239920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjdwm\" (UniqueName: \"kubernetes.io/projected/7596f7ca-55e6-4a9f-9b64-8e887451da62-kube-api-access-sjdwm\") pod \"e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") " pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.361351 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.361284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:22.637067 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:22.637024    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"]
Sep 14 03:30:22.639134 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:30:22.639079    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7596f7ca_55e6_4a9f_9b64_8e887451da62.slice/crio-618f5042b3afaa3204e41dafc0ea9d9577009a8ae2dc68a5f9e0081b071093df WatchSource:0}: Error finding container 618f5042b3afaa3204e41dafc0ea9d9577009a8ae2dc68a5f9e0081b071093df: Status 404 returned error can't find the container with id 618f5042b3afaa3204e41dafc0ea9d9577009a8ae2dc68a5f9e0081b071093df
Sep 14 03:30:23.576864 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:23.576828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj" event={"ID":"7596f7ca-55e6-4a9f-9b64-8e887451da62","Type":"ContainerStarted","Data":"f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f"}
Sep 14 03:30:23.576864 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:23.576871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj" event={"ID":"7596f7ca-55e6-4a9f-9b64-8e887451da62","Type":"ContainerStarted","Data":"618f5042b3afaa3204e41dafc0ea9d9577009a8ae2dc68a5f9e0081b071093df"}
Sep 14 03:30:23.593299 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:23.593248    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj" podStartSLOduration=1.593233211 podStartE2EDuration="1.593233211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:30:23.592298295 +0000 UTC m=+2290.318256348" watchObservedRunningTime="2026-09-14 03:30:23.593233211 +0000 UTC m=+2290.319191262"
Sep 14 03:30:27.986868 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:27.986822    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:27.987409 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:27.986881    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:27.998907 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:27.998886    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:28.617355 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:28.617323    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:30:29.549965 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:29.549925    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:30:32.362203 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:32.362167    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:32.362564 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:32.362215    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:32.374330 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:32.374306    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:30:32.640041 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:30:32.639949    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:31:05.367891 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:05.367856    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"]
Sep 14 03:31:07.442635 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:07.442558    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"]
Sep 14 03:31:07.671054 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:07.671010    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:31:15.347715 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:15.347673    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:31:20.386895 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.386836    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q" podUID="b18a09b6-e52c-4195-a06d-1537c3ded8bc" containerName="main" containerID="cri-o://145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a" gracePeriod=45
Sep 14 03:31:20.629282 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.629260    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:31:20.713285 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.713217    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tmp-dir\") pod \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") "
Sep 14 03:31:20.713444 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.713286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tls-certs\") pod \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") "
Sep 14 03:31:20.713444 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.713309    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-home\") pod \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") "
Sep 14 03:31:20.713444 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.713328    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-model-cache\") pod \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") "
Sep 14 03:31:20.713444 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.713366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn7dv\" (UniqueName: \"kubernetes.io/projected/b18a09b6-e52c-4195-a06d-1537c3ded8bc-kube-api-access-xn7dv\") pod \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") "
Sep 14 03:31:20.713444 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.713404    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-dshm\") pod \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\" (UID: \"b18a09b6-e52c-4195-a06d-1537c3ded8bc\") "
Sep 14 03:31:20.713705 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.713507    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b18a09b6-e52c-4195-a06d-1537c3ded8bc" (UID: "b18a09b6-e52c-4195-a06d-1537c3ded8bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:31:20.713705 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.713579    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-home" (OuterVolumeSpecName: "home") pod "b18a09b6-e52c-4195-a06d-1537c3ded8bc" (UID: "b18a09b6-e52c-4195-a06d-1537c3ded8bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:31:20.713813 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.713710    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:20.713813 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.713729    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:20.713813 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.713797    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "b18a09b6-e52c-4195-a06d-1537c3ded8bc" (UID: "b18a09b6-e52c-4195-a06d-1537c3ded8bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:31:20.715638 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.715614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b18a09b6-e52c-4195-a06d-1537c3ded8bc" (UID: "b18a09b6-e52c-4195-a06d-1537c3ded8bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:31:20.715895 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.715873    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-dshm" (OuterVolumeSpecName: "dshm") pod "b18a09b6-e52c-4195-a06d-1537c3ded8bc" (UID: "b18a09b6-e52c-4195-a06d-1537c3ded8bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:31:20.715971 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.715921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18a09b6-e52c-4195-a06d-1537c3ded8bc-kube-api-access-xn7dv" (OuterVolumeSpecName: "kube-api-access-xn7dv") pod "b18a09b6-e52c-4195-a06d-1537c3ded8bc" (UID: "b18a09b6-e52c-4195-a06d-1537c3ded8bc"). InnerVolumeSpecName "kube-api-access-xn7dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:31:20.814690 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.814663    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b18a09b6-e52c-4195-a06d-1537c3ded8bc-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:20.814690 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.814687    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:20.814826 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.814700    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn7dv\" (UniqueName: \"kubernetes.io/projected/b18a09b6-e52c-4195-a06d-1537c3ded8bc-kube-api-access-xn7dv\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:20.814826 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.814713    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b18a09b6-e52c-4195-a06d-1537c3ded8bc-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:20.854459 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.854434    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:31:20.881196 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.881160    2580 generic.go:356] "Generic (PLEG): container finished" podID="b18a09b6-e52c-4195-a06d-1537c3ded8bc" containerID="145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a" exitCode=0
Sep 14 03:31:20.881325 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.881201    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"
Sep 14 03:31:20.881325 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.881221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q" event={"ID":"b18a09b6-e52c-4195-a06d-1537c3ded8bc","Type":"ContainerDied","Data":"145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a"}
Sep 14 03:31:20.881325 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.881256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q" event={"ID":"b18a09b6-e52c-4195-a06d-1537c3ded8bc","Type":"ContainerDied","Data":"fee77e2fda013640efe2799a5e5901d1ceb2ca5dd859ad88b78bc21c9d30ea9a"}
Sep 14 03:31:20.881325 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.881272    2580 scope.go:117] "RemoveContainer" containerID="145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a"
Sep 14 03:31:20.893895 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.893847    2580 scope.go:117] "RemoveContainer" containerID="145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a"
Sep 14 03:31:20.895592 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:31:20.894507    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a\": container with ID starting with 145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a not found: ID does not exist" containerID="145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a"
Sep 14 03:31:20.895592 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.894546    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a"} err="failed to get container status \"145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a\": rpc error: code = NotFound desc = could not find container \"145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a\": container with ID starting with 145328085f25491cefdabdb616d4585e034adb760204172d741a3830f845746a not found: ID does not exist"
Sep 14 03:31:20.907500 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.907472    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"]
Sep 14 03:31:20.913324 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:20.913302    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-67807548-kserve-5dc4cdc75f-h2g5q"]
Sep 14 03:31:21.756321 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:21.756285    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b18a09b6-e52c-4195-a06d-1537c3ded8bc" path="/var/lib/kubelet/pods/b18a09b6-e52c-4195-a06d-1537c3ded8bc/volumes"
Sep 14 03:31:22.461046 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.460991    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj" podUID="7596f7ca-55e6-4a9f-9b64-8e887451da62" containerName="main" containerID="cri-o://f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f" gracePeriod=45
Sep 14 03:31:22.707155 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.707130    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:31:22.730965 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.730907    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjdwm\" (UniqueName: \"kubernetes.io/projected/7596f7ca-55e6-4a9f-9b64-8e887451da62-kube-api-access-sjdwm\") pod \"7596f7ca-55e6-4a9f-9b64-8e887451da62\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") "
Sep 14 03:31:22.731119 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.730965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-model-cache\") pod \"7596f7ca-55e6-4a9f-9b64-8e887451da62\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") "
Sep 14 03:31:22.731119 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.730990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-home\") pod \"7596f7ca-55e6-4a9f-9b64-8e887451da62\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") "
Sep 14 03:31:22.731119 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.731013    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7596f7ca-55e6-4a9f-9b64-8e887451da62-tls-certs\") pod \"7596f7ca-55e6-4a9f-9b64-8e887451da62\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") "
Sep 14 03:31:22.731119 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.731060    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-dshm\") pod \"7596f7ca-55e6-4a9f-9b64-8e887451da62\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") "
Sep 14 03:31:22.731119 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.731097    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-tmp-dir\") pod \"7596f7ca-55e6-4a9f-9b64-8e887451da62\" (UID: \"7596f7ca-55e6-4a9f-9b64-8e887451da62\") "
Sep 14 03:31:22.731405 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.731228    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-model-cache" (OuterVolumeSpecName: "model-cache") pod "7596f7ca-55e6-4a9f-9b64-8e887451da62" (UID: "7596f7ca-55e6-4a9f-9b64-8e887451da62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:31:22.731405 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.731249    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-home" (OuterVolumeSpecName: "home") pod "7596f7ca-55e6-4a9f-9b64-8e887451da62" (UID: "7596f7ca-55e6-4a9f-9b64-8e887451da62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:31:22.731513 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.731433    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7596f7ca-55e6-4a9f-9b64-8e887451da62" (UID: "7596f7ca-55e6-4a9f-9b64-8e887451da62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:31:22.731513 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.731481    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:22.731513 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.731499    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:22.733141 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.733113    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7596f7ca-55e6-4a9f-9b64-8e887451da62-kube-api-access-sjdwm" (OuterVolumeSpecName: "kube-api-access-sjdwm") pod "7596f7ca-55e6-4a9f-9b64-8e887451da62" (UID: "7596f7ca-55e6-4a9f-9b64-8e887451da62"). InnerVolumeSpecName "kube-api-access-sjdwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:31:22.733249 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.733157    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7596f7ca-55e6-4a9f-9b64-8e887451da62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7596f7ca-55e6-4a9f-9b64-8e887451da62" (UID: "7596f7ca-55e6-4a9f-9b64-8e887451da62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:31:22.733249 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.733198    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-dshm" (OuterVolumeSpecName: "dshm") pod "7596f7ca-55e6-4a9f-9b64-8e887451da62" (UID: "7596f7ca-55e6-4a9f-9b64-8e887451da62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:31:22.832060 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.832018    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjdwm\" (UniqueName: \"kubernetes.io/projected/7596f7ca-55e6-4a9f-9b64-8e887451da62-kube-api-access-sjdwm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:22.832060 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.832059    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7596f7ca-55e6-4a9f-9b64-8e887451da62-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:22.832392 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.832070    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:22.832392 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.832079    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7596f7ca-55e6-4a9f-9b64-8e887451da62-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\""
Sep 14 03:31:22.894177 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.894148    2580 generic.go:356] "Generic (PLEG): container finished" podID="7596f7ca-55e6-4a9f-9b64-8e887451da62" containerID="f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f" exitCode=0
Sep 14 03:31:22.894272 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.894193    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"
Sep 14 03:31:22.894272 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.894234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj" event={"ID":"7596f7ca-55e6-4a9f-9b64-8e887451da62","Type":"ContainerDied","Data":"f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f"}
Sep 14 03:31:22.894355 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.894276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj" event={"ID":"7596f7ca-55e6-4a9f-9b64-8e887451da62","Type":"ContainerDied","Data":"618f5042b3afaa3204e41dafc0ea9d9577009a8ae2dc68a5f9e0081b071093df"}
Sep 14 03:31:22.894355 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.894297    2580 scope.go:117] "RemoveContainer" containerID="f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f"
Sep 14 03:31:22.903873 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.903858    2580 scope.go:117] "RemoveContainer" containerID="f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f"
Sep 14 03:31:22.904147 ip-10-0-138-23 kubenswrapper[2580]: E0914 03:31:22.904125    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f\": container with ID starting with f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f not found: ID does not exist" containerID="f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f"
Sep 14 03:31:22.904254 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.904156    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f"} err="failed to get container status \"f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f\": rpc error: code = NotFound desc = could not find container \"f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f\": container with ID starting with f9a6627e1ba1f2753cbaf11cb80c225d8c91d06751778a5aa6b813e2c4933f5f not found: ID does not exist"
Sep 14 03:31:22.916200 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.916180    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"]
Sep 14 03:31:22.918253 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:22.918235    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj"]
Sep 14 03:31:23.757843 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:23.757813    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7596f7ca-55e6-4a9f-9b64-8e887451da62" path="/var/lib/kubelet/pods/7596f7ca-55e6-4a9f-9b64-8e887451da62/volumes"
Sep 14 03:31:27.630208 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:27.630166    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-67807548-kserve-f785d47c5-v5bjj" podUID="7596f7ca-55e6-4a9f-9b64-8e887451da62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Sep 14 03:31:28.245560 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.245519    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:31:28.445483 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.445455    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-559c5f7949-n7824"]
Sep 14 03:31:28.445802 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.445790    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18a09b6-e52c-4195-a06d-1537c3ded8bc" containerName="main"
Sep 14 03:31:28.445851 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.445803    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18a09b6-e52c-4195-a06d-1537c3ded8bc" containerName="main"
Sep 14 03:31:28.445851 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.445820    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7596f7ca-55e6-4a9f-9b64-8e887451da62" containerName="main"
Sep 14 03:31:28.445851 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.445825    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7596f7ca-55e6-4a9f-9b64-8e887451da62" containerName="main"
Sep 14 03:31:28.445948 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.445882    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7596f7ca-55e6-4a9f-9b64-8e887451da62" containerName="main"
Sep 14 03:31:28.445948 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.445893    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18a09b6-e52c-4195-a06d-1537c3ded8bc" containerName="main"
Sep 14 03:31:28.450048 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.450017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.452461 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.452436    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:31:28.453134 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.453114    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jf78h\""
Sep 14 03:31:28.453250 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.453112    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 03:31:28.453317 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.453116    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 14 03:31:28.453382 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.453119    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 03:31:28.461663 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.461639    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-559c5f7949-n7824"]
Sep 14 03:31:28.480118 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.480092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e2fa7b-4d42-417b-9624-d4359c953644-openshift-service-ca\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.480232 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.480129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e2fa7b-4d42-417b-9624-d4359c953644-metrics-certs\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.480232 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.480152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rw4l\" (UniqueName: \"kubernetes.io/projected/88e2fa7b-4d42-417b-9624-d4359c953644-kube-api-access-7rw4l\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.480360 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.480285    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e2fa7b-4d42-417b-9624-d4359c953644-maas-api-tls\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.580824 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.580801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e2fa7b-4d42-417b-9624-d4359c953644-maas-api-tls\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.580949 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.580842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e2fa7b-4d42-417b-9624-d4359c953644-openshift-service-ca\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.580949 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.580868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e2fa7b-4d42-417b-9624-d4359c953644-metrics-certs\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.581093 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.580985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rw4l\" (UniqueName: \"kubernetes.io/projected/88e2fa7b-4d42-417b-9624-d4359c953644-kube-api-access-7rw4l\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.581477 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.581454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e2fa7b-4d42-417b-9624-d4359c953644-openshift-service-ca\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.583231 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.583204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e2fa7b-4d42-417b-9624-d4359c953644-metrics-certs\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.583382 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.583362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e2fa7b-4d42-417b-9624-d4359c953644-maas-api-tls\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.588092 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.588074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rw4l\" (UniqueName: \"kubernetes.io/projected/88e2fa7b-4d42-417b-9624-d4359c953644-kube-api-access-7rw4l\") pod \"maas-api-559c5f7949-n7824\" (UID: \"88e2fa7b-4d42-417b-9624-d4359c953644\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:28.762774 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:28.762738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:29.052156 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:29.052127    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-559c5f7949-n7824"]
Sep 14 03:31:29.055194 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:31:29.055162    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e2fa7b_4d42_417b_9624_d4359c953644.slice/crio-ee9d0a6363c370973ca3b2fff48bf6c6b745781c452710583bf8d68a73d61815 WatchSource:0}: Error finding container ee9d0a6363c370973ca3b2fff48bf6c6b745781c452710583bf8d68a73d61815: Status 404 returned error can't find the container with id ee9d0a6363c370973ca3b2fff48bf6c6b745781c452710583bf8d68a73d61815
Sep 14 03:31:29.057361 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:29.057334    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 03:31:29.934173 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:29.934142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824" event={"ID":"88e2fa7b-4d42-417b-9624-d4359c953644","Type":"ContainerStarted","Data":"7542d94a48cd54499ce64c2cddaaf6a22883e56599d68bf90fca02533e86e0f4"}
Sep 14 03:31:29.934546 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:29.934195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824" event={"ID":"88e2fa7b-4d42-417b-9624-d4359c953644","Type":"ContainerStarted","Data":"ee9d0a6363c370973ca3b2fff48bf6c6b745781c452710583bf8d68a73d61815"}
Sep 14 03:31:29.934546 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:29.934233    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:29.950025 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:29.949971    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824" podStartSLOduration=1.585560305 podStartE2EDuration="1.949956097s" totalImagesPullingTime="364.395792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:31:29.948814287 +0000 UTC m=+2356.674772341" watchObservedRunningTime="2026-09-14 03:31:29.949956097 +0000 UTC m=+2356.675914149"
Sep 14 03:31:35.067548 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:35.067513    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:31:35.944833 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:35.944804    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-559c5f7949-n7824"
Sep 14 03:31:40.674410 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:40.674371    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:31:46.355611 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:46.355573    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:31:53.176727 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:31:53.176692    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-grwpz"]
Sep 14 03:32:02.277775 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:02.277739    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mrvg2_a127e29d-b809-4399-a221-c740bfb18d64/manager/0.log"
Sep 14 03:32:02.711302 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:02.711231    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mmsrq_49362469-29e3-4602-b79a-96998c748d9b/manager/0.log"
Sep 14 03:32:03.111682 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:03.111652    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ffb7699c-crjkz_6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e/copy-manifests/0.log"
Sep 14 03:32:03.123997 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:03.123972    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ffb7699c-crjkz_6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e/manager/0.log"
Sep 14 03:32:03.597590 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:03.597559    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-559c5f7949-n7824_88e2fa7b-4d42-417b-9624-d4359c953644/maas-api/0.log"
Sep 14 03:32:05.156795 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:05.156767    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bvpp2_ae1db549-83b6-437e-ac63-0e4f4c97127f/kuadrant-console-plugin/0.log"
Sep 14 03:32:05.455081 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:05.454978    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-grwpz_0f00d0aa-4331-4c73-aa5f-553943c9ada2/limitador/0.log"
Sep 14 03:32:05.602343 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:05.602313    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-68nng_1838aa69-6c78-4cde-9036-54771d072762/manager/0.log"
Sep 14 03:32:05.976890 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:05.976860    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6_29fb8508-202c-45bd-a05b-010246a3bec1/istio-proxy/0.log"
Sep 14 03:32:06.478810 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:06.472674    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-bh7f6_65744834-d235-4b52-b655-6b0c87e69c27/istio-proxy/1.log"
Sep 14 03:32:06.873330 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:06.873298    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b98ff7586-7xqbj_4a5a2f9a-1382-4c95-a2d7-ab0484564a3c/router/0.log"
Sep 14 03:32:07.894868 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:07.894833    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-r9fpv_405ebb2d-d206-43d7-a359-8130282f0a6c/main/0.log"
Sep 14 03:32:08.011347 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:08.011318    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-jqjrf_17dde9f1-556f-49f9-a355-dbe8826bbc19/main/0.log"
Sep 14 03:32:13.926224 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:13.926187    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f2kzn/must-gather-mp7fv"]
Sep 14 03:32:13.930081 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:13.930062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2kzn/must-gather-mp7fv"
Sep 14 03:32:13.932399 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:13.932356    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2kzn\"/\"openshift-service-ca.crt\""
Sep 14 03:32:13.932552 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:13.932531    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f2kzn\"/\"default-dockercfg-v4n8b\""
Sep 14 03:32:13.932806 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:13.932789    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2kzn\"/\"kube-root-ca.crt\""
Sep 14 03:32:13.961434 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:13.961408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnk46\" (UniqueName: \"kubernetes.io/projected/e7df9ec0-beb1-4fb6-8251-5bb6410b5068-kube-api-access-qnk46\") pod \"must-gather-mp7fv\" (UID: \"e7df9ec0-beb1-4fb6-8251-5bb6410b5068\") " pod="openshift-must-gather-f2kzn/must-gather-mp7fv"
Sep 14 03:32:13.961661 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:13.961617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7df9ec0-beb1-4fb6-8251-5bb6410b5068-must-gather-output\") pod \"must-gather-mp7fv\" (UID: \"e7df9ec0-beb1-4fb6-8251-5bb6410b5068\") " pod="openshift-must-gather-f2kzn/must-gather-mp7fv"
Sep 14 03:32:13.972341 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:13.972315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2kzn/must-gather-mp7fv"]
Sep 14 03:32:13.979682 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:13.979658    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:32:13.980591 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:13.980576    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:32:14.062156 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:14.062135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnk46\" (UniqueName: \"kubernetes.io/projected/e7df9ec0-beb1-4fb6-8251-5bb6410b5068-kube-api-access-qnk46\") pod \"must-gather-mp7fv\" (UID: \"e7df9ec0-beb1-4fb6-8251-5bb6410b5068\") " pod="openshift-must-gather-f2kzn/must-gather-mp7fv"
Sep 14 03:32:14.062284 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:14.062173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7df9ec0-beb1-4fb6-8251-5bb6410b5068-must-gather-output\") pod \"must-gather-mp7fv\" (UID: \"e7df9ec0-beb1-4fb6-8251-5bb6410b5068\") " pod="openshift-must-gather-f2kzn/must-gather-mp7fv"
Sep 14 03:32:14.062456 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:14.062439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7df9ec0-beb1-4fb6-8251-5bb6410b5068-must-gather-output\") pod \"must-gather-mp7fv\" (UID: \"e7df9ec0-beb1-4fb6-8251-5bb6410b5068\") " pod="openshift-must-gather-f2kzn/must-gather-mp7fv"
Sep 14 03:32:14.080267 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:14.080247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnk46\" (UniqueName: \"kubernetes.io/projected/e7df9ec0-beb1-4fb6-8251-5bb6410b5068-kube-api-access-qnk46\") pod \"must-gather-mp7fv\" (UID: \"e7df9ec0-beb1-4fb6-8251-5bb6410b5068\") " pod="openshift-must-gather-f2kzn/must-gather-mp7fv"
Sep 14 03:32:14.238305 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:14.238238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2kzn/must-gather-mp7fv"
Sep 14 03:32:14.287635 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:14.287480    2580 scope.go:117] "RemoveContainer" containerID="d7af5bf23d488939b2521ac75c3b87dd704defceb604bc52773320cec8e99be0"
Sep 14 03:32:14.531493 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:14.531469    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2kzn/must-gather-mp7fv"]
Sep 14 03:32:14.533244 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:32:14.533217    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7df9ec0_beb1_4fb6_8251_5bb6410b5068.slice/crio-4b36fe6cb9e0773d2dcad7dbca8e03b61211f380f8758267c9829b5b2ffe7344 WatchSource:0}: Error finding container 4b36fe6cb9e0773d2dcad7dbca8e03b61211f380f8758267c9829b5b2ffe7344: Status 404 returned error can't find the container with id 4b36fe6cb9e0773d2dcad7dbca8e03b61211f380f8758267c9829b5b2ffe7344
Sep 14 03:32:15.172488 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:15.172451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2kzn/must-gather-mp7fv" event={"ID":"e7df9ec0-beb1-4fb6-8251-5bb6410b5068","Type":"ContainerStarted","Data":"4b36fe6cb9e0773d2dcad7dbca8e03b61211f380f8758267c9829b5b2ffe7344"}
Sep 14 03:32:16.185788 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:16.185753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2kzn/must-gather-mp7fv" event={"ID":"e7df9ec0-beb1-4fb6-8251-5bb6410b5068","Type":"ContainerStarted","Data":"591960189aeeefe5846194309cf3666d3bbe20e981fafdd25012fb1313291ae4"}
Sep 14 03:32:16.186197 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:16.185808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2kzn/must-gather-mp7fv" event={"ID":"e7df9ec0-beb1-4fb6-8251-5bb6410b5068","Type":"ContainerStarted","Data":"a095ff1dbad05bdb7e822a093f7c17330810785a2fdf31766d296954e9b5aba4"}
Sep 14 03:32:16.217070 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:16.217001    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f2kzn/must-gather-mp7fv" podStartSLOduration=2.428383914 podStartE2EDuration="3.216984477s" totalImagesPullingTime="788.600563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:32:16.214828967 +0000 UTC m=+2402.940787023" watchObservedRunningTime="2026-09-14 03:32:16.216984477 +0000 UTC m=+2402.942942528"
Sep 14 03:32:17.457107 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:17.457079    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ggtgm_75b4ab6d-419f-495a-b2f5-d62971a327d8/global-pull-secret-syncer/0.log"
Sep 14 03:32:17.853386 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:17.853360    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-thtkr_b0d4b577-26c7-447d-b6ed-5834730b4051/konnectivity-agent/0.log"
Sep 14 03:32:17.896940 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:17.896912    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-23.ec2.internal_5480c88e36bfd2aebb919a8fa1692c84/haproxy/0.log"
Sep 14 03:32:22.931808 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:22.931775    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bvpp2_ae1db549-83b6-437e-ac63-0e4f4c97127f/kuadrant-console-plugin/0.log"
Sep 14 03:32:23.085408 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:23.085376    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-grwpz_0f00d0aa-4331-4c73-aa5f-553943c9ada2/limitador/0.log"
Sep 14 03:32:23.169585 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:23.169555    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-68nng_1838aa69-6c78-4cde-9036-54771d072762/manager/0.log"
Sep 14 03:32:25.214552 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:25.214499    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-8gcdn_91e84826-1e48-496a-9617-1bf6cf76c0e4/cluster-monitoring-operator/0.log"
Sep 14 03:32:25.750024 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:25.749989    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xmkc6_dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b/node-exporter/0.log"
Sep 14 03:32:25.790205 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:25.790162    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xmkc6_dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b/kube-rbac-proxy/0.log"
Sep 14 03:32:25.830187 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:25.830162    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xmkc6_dfd2c10a-4113-42c7-88b2-3dfa8bad9d7b/init-textfile/0.log"
Sep 14 03:32:27.052915 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.052879    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"]
Sep 14 03:32:27.055807 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.055782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.072210 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.072182    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"]
Sep 14 03:32:27.201659 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.201608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-lib-modules\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.201835 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.201664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhd8x\" (UniqueName: \"kubernetes.io/projected/2c6ce494-6fa7-4a0b-892a-a5012024307f-kube-api-access-jhd8x\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.201835 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.201766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-proc\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.201835 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.201800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-sys\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.201835 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.201834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-podres\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.303336 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.303249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-sys\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.303336 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.303299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-podres\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.303579 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.303372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-lib-modules\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.303579 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.303377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-sys\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.303579 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.303397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhd8x\" (UniqueName: \"kubernetes.io/projected/2c6ce494-6fa7-4a0b-892a-a5012024307f-kube-api-access-jhd8x\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.303579 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.303461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-proc\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.303579 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.303488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-lib-modules\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.303579 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.303497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-podres\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.303579 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.303543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c6ce494-6fa7-4a0b-892a-a5012024307f-proc\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.314352 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.314319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhd8x\" (UniqueName: \"kubernetes.io/projected/2c6ce494-6fa7-4a0b-892a-a5012024307f-kube-api-access-jhd8x\") pod \"perf-node-gather-daemonset-h8jqz\" (UID: \"2c6ce494-6fa7-4a0b-892a-a5012024307f\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.368641 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.368608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:27.673606 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:27.673530    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"]
Sep 14 03:32:27.677655 ip-10-0-138-23 kubenswrapper[2580]: W0914 03:32:27.677610    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2c6ce494_6fa7_4a0b_892a_a5012024307f.slice/crio-da27c35dd221c56ac7808bdd31edc7229a6db0836925b84e2f02364cdbb999c1 WatchSource:0}: Error finding container da27c35dd221c56ac7808bdd31edc7229a6db0836925b84e2f02364cdbb999c1: Status 404 returned error can't find the container with id da27c35dd221c56ac7808bdd31edc7229a6db0836925b84e2f02364cdbb999c1
Sep 14 03:32:28.286936 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:28.286899    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz" event={"ID":"2c6ce494-6fa7-4a0b-892a-a5012024307f","Type":"ContainerStarted","Data":"79e482b889a54f71b6820e371cf1643db002f704dd6c39131bc74f7052c1fcb1"}
Sep 14 03:32:28.287538 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:28.287509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz" event={"ID":"2c6ce494-6fa7-4a0b-892a-a5012024307f","Type":"ContainerStarted","Data":"da27c35dd221c56ac7808bdd31edc7229a6db0836925b84e2f02364cdbb999c1"}
Sep 14 03:32:28.288462 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:28.288440    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:28.315863 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:28.315818    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz" podStartSLOduration=1.315804735 podStartE2EDuration="1.315804735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 03:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:32:28.315147039 +0000 UTC m=+2415.041105092" watchObservedRunningTime="2026-09-14 03:32:28.315804735 +0000 UTC m=+2415.041762786"
Sep 14 03:32:28.691958 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:28.691876    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/1.log"
Sep 14 03:32:28.696570 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:28.696546    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-h97tj_39b540b4-971c-4b5e-9479-1a4a60307be1/console-operator/2.log"
Sep 14 03:32:29.183100 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:29.183068    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-666bf7cc8f-v7qng_98c6085d-e5f9-43c4-be8f-a0c0a683f91d/console/0.log"
Sep 14 03:32:29.235046 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:29.235005    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-p7bpw_d642bccd-dc72-4526-8304-fa4f74691d91/download-server/0.log"
Sep 14 03:32:29.884916 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:29.884889    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-8gp2j_5283b251-dd1e-4ff7-88f4-76b8dd5273ee/volume-data-source-validator/0.log"
Sep 14 03:32:31.027978 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:31.027952    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lqmmv_799a37d7-2bc7-44ff-bd8d-4ddb79b88a38/dns/0.log"
Sep 14 03:32:31.051336 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:31.051289    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lqmmv_799a37d7-2bc7-44ff-bd8d-4ddb79b88a38/kube-rbac-proxy/0.log"
Sep 14 03:32:31.207394 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:31.207367    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6ps5c_c3944957-e448-4279-834a-1b091a14afe8/dns-node-resolver/0.log"
Sep 14 03:32:32.012111 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:32.012083    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-852l7_6d2fa0b7-cc69-4546-b798-f5114afde64a/node-ca/0.log"
Sep 14 03:32:33.245723 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:33.245679    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6_29fb8508-202c-45bd-a05b-010246a3bec1/istio-proxy/0.log"
Sep 14 03:32:33.787153 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:33.787121    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-bh7f6_65744834-d235-4b52-b655-6b0c87e69c27/istio-proxy/0.log"
Sep 14 03:32:33.790153 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:33.790127    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-bh7f6_65744834-d235-4b52-b655-6b0c87e69c27/istio-proxy/1.log"
Sep 14 03:32:33.944827 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:33.944796    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b98ff7586-7xqbj_4a5a2f9a-1382-4c95-a2d7-ab0484564a3c/router/0.log"
Sep 14 03:32:35.010653 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:35.010623    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hzbqh_34f503e3-ca4e-4080-9bd6-0e343b5f1b90/serve-healthcheck-canary/0.log"
Sep 14 03:32:35.311101 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:35.311007    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-h8jqz"
Sep 14 03:32:35.649337 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:35.649250    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-kmjvv_1a167d62-c256-4cc5-a3e6-5f6eb19cdb15/insights-operator/0.log"
Sep 14 03:32:35.650062 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:35.650042    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-kmjvv_1a167d62-c256-4cc5-a3e6-5f6eb19cdb15/insights-operator/1.log"
Sep 14 03:32:35.912041 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:35.911961    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgmx4_fb5579be-c3dc-43f1-aa7c-3bd709206921/kube-rbac-proxy/0.log"
Sep 14 03:32:35.944615 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:35.944587    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgmx4_fb5579be-c3dc-43f1-aa7c-3bd709206921/exporter/0.log"
Sep 14 03:32:35.976892 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:35.976863    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgmx4_fb5579be-c3dc-43f1-aa7c-3bd709206921/extractor/0.log"
Sep 14 03:32:37.254892 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:37.254861    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6_29fb8508-202c-45bd-a05b-010246a3bec1/istio-proxy/0.log"
Sep 14 03:32:37.747164 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:37.747133    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-bh7f6_65744834-d235-4b52-b655-6b0c87e69c27/istio-proxy/0.log"
Sep 14 03:32:37.749964 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:37.749941    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-bh7f6_65744834-d235-4b52-b655-6b0c87e69c27/istio-proxy/1.log"
Sep 14 03:32:37.878114 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:37.878085    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b98ff7586-7xqbj_4a5a2f9a-1382-4c95-a2d7-ab0484564a3c/router/0.log"
Sep 14 03:32:38.969407 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:38.969313    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mrvg2_a127e29d-b809-4399-a221-c740bfb18d64/manager/0.log"
Sep 14 03:32:39.176514 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:39.176481    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mmsrq_49362469-29e3-4602-b79a-96998c748d9b/manager/0.log"
Sep 14 03:32:39.374764 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:39.374730    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ffb7699c-crjkz_6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e/manager/0.log"
Sep 14 03:32:39.401014 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:39.400986    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ffb7699c-crjkz_6f3f0a6c-9145-49f1-b4e6-a2ead4d3f75e/copy-manifests/0.log"
Sep 14 03:32:39.513013 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:39.512972    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdbzt6_29fb8508-202c-45bd-a05b-010246a3bec1/istio-proxy/0.log"
Sep 14 03:32:40.059624 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:40.059546    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-bh7f6_65744834-d235-4b52-b655-6b0c87e69c27/istio-proxy/0.log"
Sep 14 03:32:40.061806 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:40.061741    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-bh7f6_65744834-d235-4b52-b655-6b0c87e69c27/istio-proxy/1.log"
Sep 14 03:32:40.158903 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:40.158872    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b98ff7586-7xqbj_4a5a2f9a-1382-4c95-a2d7-ab0484564a3c/router/0.log"
Sep 14 03:32:40.779591 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:40.779564    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59b9ccfbf8-trr68_959c47ab-a1ab-4a6b-ad95-aeb95acd3e8a/manager/0.log"
Sep 14 03:32:40.856368 ip-10-0-138-23 kubenswrapper[2580]: I0914 03:32:40.856334    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-nf6l4_96c350f8-214a-4c63-bfed-89d6929be48a/openshift-lws-operator/0.log"