Sep 11 18:13:47.013456 ip-10-0-143-92 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 18:13:47.013473 ip-10-0-143-92 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 18:13:47.013483 ip-10-0-143-92 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 18:13:47.013682 ip-10-0-143-92 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 18:13:58.374777 ip-10-0-143-92 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 18:13:58.374795 ip-10-0-143-92 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c507f165c6374a47b2621a54f9817f88 --
Sep 11 18:15:43.499353 ip-10-0-143-92 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 18:15:43.914579 ip-10-0-143-92 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:43.914579 ip-10-0-143-92 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 18:15:43.914579 ip-10-0-143-92 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:43.914579 ip-10-0-143-92 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 18:15:43.914579 ip-10-0-143-92 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:43.915757 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.915689    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 18:15:43.919760 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919747    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919761    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919765    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919770    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919774    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919777    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919780    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919783    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919786    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919789    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919792    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919794    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919797    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919800    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:43.919795 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919802    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919805    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919808    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919810    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919813    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919816    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919819    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919821    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919824    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919830    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919832    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919835    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919837    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919840    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919842    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919845    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919847    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919850    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919853    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919856    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:43.920139 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919858    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919861    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919863    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919866    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919868    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919871    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919873    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919876    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919878    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919881    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919883    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919886    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919888    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919891    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919894    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919897    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919899    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919902    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919904    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919907    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:43.920654 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919909    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919912    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919916    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919920    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919923    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919925    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919928    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919930    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919933    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919935    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919938    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919940    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919943    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919945    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919948    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919952    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919955    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919959    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919961    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:43.921159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919964    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919967    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919970    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919972    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919975    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919978    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919981    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919986    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919988    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919991    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919993    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919996    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.919998    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920001    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920004    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920006    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920008    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920011    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920013    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:43.921617 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920016    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920018    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920021    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920023    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920026    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920028    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920031    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920717    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920724    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920727    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920730    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920733    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920736    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920739    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920741    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920744    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920746    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920749    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920751    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920754    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:43.922077 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920757    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920760    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920763    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920766    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920769    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920771    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920774    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920776    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920779    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920782    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920784    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920787    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920789    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920792    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920794    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920797    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920799    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920802    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920804    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920807    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:43.922596 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920809    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920812    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920814    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920817    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920819    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920822    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920824    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920827    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920829    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920831    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920834    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920836    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920840    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920843    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920846    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920848    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920851    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920853    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920856    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920859    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:43.923083 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920861    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920863    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920866    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920869    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920871    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920874    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920876    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920879    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920881    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920885    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920889    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920892    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920895    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920898    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920901    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920903    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920906    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920909    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920912    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:43.923582 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920914    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920917    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920921    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920924    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920929    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920932    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920935    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920938    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920941    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920943    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920946    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920948    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920951    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920953    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920956    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920958    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920961    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920963    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920966    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:43.924048 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920968    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920971    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920973    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920975    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920978    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920980    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920983    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.920985    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921342    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921347    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921350    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921353    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921356    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921358    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921361    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921364    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921366    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921369    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921372    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921375    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:43.924546 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921378    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921380    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921383    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921386    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921389    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921391    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921393    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921400    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921403    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921406    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921409    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921413    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921415    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921418    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921421    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921423    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921426    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921428    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921431    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:43.925040 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921433    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921435    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921438    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921441    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921443    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921445    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921448    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921450    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921453    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921455    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921458    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921461    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921463    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921466    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921469    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921472    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921475    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921477    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921480    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921482    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:43.925553 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921487    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921489    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921491    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921494    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921496    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921499    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921501    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921504    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921506    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921509    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921511    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921514    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921517    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921519    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921522    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921524    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921527    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921529    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921531    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:43.926036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921534    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921536    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921539    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921542    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921545    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921547    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921550    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921553    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921555    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921559    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921562    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921565    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921568    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921573    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921575    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921579    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921582    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921585    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921588    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:43.926524 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921590    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921593    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921596    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921598    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921601    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921604    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921606    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921609    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921612    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.921614    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921689    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921718    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921724    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921728    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921732    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921735    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921739    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921743    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921746    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921749    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921753    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921756    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 18:15:43.927015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921759    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921762    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921765    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921768    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921772    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921775    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921779    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921781    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921784    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921787    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921790    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921793    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921796    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921799    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921803    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921806    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921809    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921812    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921816    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921819    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921823    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921826    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921828    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921831    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921834    2563 flags.go:64] FLAG: --enable-server="true"
Sep 11 18:15:43.927597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921837    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921845    2563 flags.go:64] FLAG: --event-burst="100"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921849    2563 flags.go:64] FLAG: --event-qps="50"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921852    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921855    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921857    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921861    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921864    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921867    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921870    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921873    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921877    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921880    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921883    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921886    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921889    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921892    2563 flags.go:64] FLAG: --feature-gates=""
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921896    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921899    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921902    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921905    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921908    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921911    2563 flags.go:64] FLAG: --help="false"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921914    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-143-92.ec2.internal"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921917    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 18:15:43.928240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921920    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921923    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921926    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921929    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921932    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921935    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921938    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921940    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921944    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921947    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921950    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921953    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921956    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921959    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921962    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921964    2563 flags.go:64] FLAG: --lock-file=""
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921968    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921971    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921975    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921980    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921983    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921986    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921988    2563 flags.go:64] FLAG: --logging-format="text"
Sep 11 18:15:43.928871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921991    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921994    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.921997    2563 flags.go:64] FLAG: --manifest-url=""
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922001    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922004    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922007    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922011    2563 flags.go:64] FLAG: --max-pods="110"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922014    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922017    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922020    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922023    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922025    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922028    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922031    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922038    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922040    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922043    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922047    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922050    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922055    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922058    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922061    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922064    2563 flags.go:64] FLAG: --port="10250"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922067    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 18:15:43.929536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922070    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07f011c6d43857a21"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922073    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922076    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922081    2563 flags.go:64] FLAG: --register-node="true"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922096    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922100    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922103    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922106    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922109    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922113    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922116    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922119    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922122    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922124    2563 flags.go:64] FLAG: --runonce="false"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922127    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922130    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922134    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922137    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922139    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922142    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922145    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922149    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922152    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922154    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922157    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922160    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 18:15:43.930136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922165    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922168    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922170    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922176    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922178    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922181    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922185    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922187    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922190    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922194    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922197    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922200    2563 flags.go:64] FLAG: --v="2"
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922204    2563 flags.go:64] FLAG: --version="false"
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922208    2563 flags.go:64] FLAG: --vmodule=""
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922211    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.922214    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922325    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922332    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922335    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922339    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922342    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922344    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922347    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:43.930855 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922349    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922352    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922355    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922358    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922361    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922363    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922366    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922369    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922371    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922375    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922378    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922381    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922383    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922386    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922389    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922392    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922396    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922399    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922402    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:43.931431 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922405    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922408    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922410    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922413    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922416    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922418    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922421    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922423    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922426    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922429    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922431    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922434    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922436    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922439    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922442    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922445    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922447    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922450    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922452    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922455    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:43.931903 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922458    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922460    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922464    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922467    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922469    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922472    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922475    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922478    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922480    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922483    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922486    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922488    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922491    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922494    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922497    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922499    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922502    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922504    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922507    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922510    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:43.932451 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922512    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922515    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922518    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922521    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922523    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922526    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922530    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922533    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922536    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922539    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922542    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922544    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922547    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922550    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922554    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922556    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922559    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922562    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922564    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922567    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:43.932915 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922570    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922572    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922575    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922579    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922583    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922585    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922589    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922593    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922597    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922601    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922604    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922607    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.922610    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.923340    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.930418    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 18:15:43.933395 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.930432    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.931338    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.931345    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931399    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931404    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931407    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931410    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931413    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931416    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931419    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931421    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931424    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931426    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931429    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931432    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931435    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931438    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931440    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931443    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:43.933763 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931446    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931449    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931452    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931455    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931457    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931460    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931463    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931465    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931468    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931470    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931474    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931478    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931482    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931485    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931488    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931491    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931494    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931497    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931500    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931502    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:43.934245 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931505    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931519    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931522    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931525    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931527    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931530    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931532    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931534    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931537    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931540    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931542    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931545    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931547    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931550    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931552    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931555    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931557    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931560    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931563    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931565    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:43.934737 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931567    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931570    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931572    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931575    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931577    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931580    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931583    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931586    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931588    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931592    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931595    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931599    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931602    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931605    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931607    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931610    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931613    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931615    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931618    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931621    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:43.935238 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931624    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931626    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931629    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931632    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931635    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931637    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931640    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931643    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931646    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931649    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931652    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931654    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931657    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931659    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931662    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931665    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931667    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931670    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931672    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931675    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:43.935726 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931678    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931680    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931684    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.931689    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931791    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931796    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931799    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931802    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931805    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931808    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931811    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931815    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931818    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931820    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:43.936237 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931823    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931825    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931828    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931830    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931832    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931835    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931839    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931842    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931844    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931847    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931850    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931852    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931855    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931857    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931860    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931862    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931865    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931867    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931870    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:43.936589 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931873    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931875    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931878    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931880    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931883    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931885    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931888    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931890    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931893    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931895    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931898    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931900    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931903    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931906    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931909    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931911    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931914    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931917    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931919    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931922    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:43.937039 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931924    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931927    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931929    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931932    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931934    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931937    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931939    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931941    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931944    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931946    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931949    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931951    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931954    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931956    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931959    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931961    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931966    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931968    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931971    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931974    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:43.937531 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931976    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931978    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931981    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931983    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931985    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931988    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931990    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931993    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931996    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.931998    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932000    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932003    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932006    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932008    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932010    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932014    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932017    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932020    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932023    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:43.937993 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932025    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932028    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932031    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932033    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932036    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932039    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932041    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932044    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932046    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932050    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:43.932053    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.932057    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.932168    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.935400    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 18:15:43.938473 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.936559    2563 server.go:1019] "Starting client certificate rotation"
Sep 11 18:15:43.938817 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.936650    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:15:43.938817 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.936687    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:15:43.939463 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.939450    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:15:43.940466 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.940450    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:15:43.952694 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.952675    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 11 18:15:43.955801 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.955784    2563 log.go:25] "Validated CRI v1 image API"
Sep 11 18:15:43.959686 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.959663    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 18:15:43.962938 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.962913    2563 fs.go:135] Filesystem UUIDs: map[1866de9d-1d03-4aaa-b77d-886eb05fcaa9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7a27edf9-07eb-4596-b0d9-8a4c3949a8e2:/dev/nvme0n1p3]
Sep 11 18:15:43.962992 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.962932    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 18:15:43.967792 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.967689    2563 manager.go:216] Machine: {Timestamp:2026-09-11 18:15:43.966253948 +0000 UTC m=+0.365052826 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3154339 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec256dd3f42d101b12b306b3c5efc7a1 SystemUUID:ec256dd3-f42d-101b-12b3-06b3c5efc7a1 BootID:c507f165-c637-4a47-b262-1a54f9817f88 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bf:1b:d0:41:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bf:1b:d0:41:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:5d:ec:ef:6f:89 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 18:15:43.967792 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.967790    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 18:15:43.967896 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.967885    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 18:15:43.969911 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.969897    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:15:43.971429 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.971407    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 18:15:43.971567 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.971432    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 18:15:43.972256 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.972244    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 18:15:43.972288 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.972257    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 18:15:43.972288 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.972270    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:15:43.973180 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.973169    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:15:43.974415 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.974405    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 18:15:43.979820 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.979799    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:15:43.979939 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.979929    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 18:15:43.979969 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.979961    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 18:15:43.979996 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.979975    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 18:15:43.980022 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.980001    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 18:15:43.980022 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.980016    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 11 18:15:43.980080 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.980028    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 18:15:43.980404 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.980393    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:15:43.980445 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.980416    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:15:43.982809 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.982793    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 18:15:43.985005 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.984992    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 18:15:43.985047 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.985022    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 18:15:43.987761 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987749    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 18:15:43.987805 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987767    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 18:15:43.987805 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987773    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 18:15:43.987805 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987778    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 18:15:43.987805 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987786    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 18:15:43.987805 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987797    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 18:15:43.987950 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987810    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 18:15:43.987950 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987815    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 18:15:43.987950 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987821    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 18:15:43.987950 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987826    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 18:15:43.987950 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987840    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 18:15:43.987950 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.987845    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 18:15:43.988786 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.988776    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 18:15:43.988786 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.988786    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 18:15:43.992401 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.992381    2563 server.go:1268] "Started kubelet"
Sep 11 18:15:43.992491 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.992380    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-92.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 18:15:43.992491 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.992442    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 18:15:43.992563 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.992472    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 18:15:43.992563 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.992528    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 18:15:43.992769 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:43.992729    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 18:15:43.992955 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:43.992932    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 18:15:43.993142 ip-10-0-143-92 systemd[1]: Started Kubernetes Kubelet.
Sep 11 18:15:43.998035 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.998006    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 18:15:43.998902 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:43.998871    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 18:15:44.000237 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.000202    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-scq2x"
Sep 11 18:15:44.000573 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.000529    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 18:15:44.004625 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.004607    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 18:15:44.004846 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.003701    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-92.ec2.internal.18d456f0c05c82a0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-92.ec2.internal,UID:ip-10-0-143-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-92.ec2.internal,},FirstTimestamp:2026-09-11 18:15:43.992357536 +0000 UTC m=+0.391156395,LastTimestamp:2026-09-11 18:15:43.992357536 +0000 UTC m=+0.391156395,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-92.ec2.internal,}"
Sep 11 18:15:44.005244 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.005227    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 18:15:44.005372 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.005353    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-92.ec2.internal\" not found"
Sep 11 18:15:44.005994 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.005979    2563 factory.go:55] Registering systemd factory
Sep 11 18:15:44.005994 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.005994    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 11 18:15:44.006124 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.006107    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 18:15:44.006170 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.006127    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 18:15:44.006170 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.006111    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 18:15:44.006250 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.006192    2563 factory.go:153] Registering CRI-O factory
Sep 11 18:15:44.006250 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.006207    2563 factory.go:223] Registration of the crio container factory successfully
Sep 11 18:15:44.006250 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.006220    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 18:15:44.006250 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.006229    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 18:15:44.006250 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.006245    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 18:15:44.006484 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.006259    2563 factory.go:103] Registering Raw factory
Sep 11 18:15:44.006484 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.006269    2563 manager.go:1196] Started watching for new ooms in manager
Sep 11 18:15:44.006636 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.006624    2563 manager.go:349] Starting recovery of all containers
Sep 11 18:15:44.009009 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.008986    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-scq2x"
Sep 11 18:15:44.012942 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.012915    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 18:15:44.013030 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.012986    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 18:15:44.017292 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.017276    2563 manager.go:354] Recovery completed
Sep 11 18:15:44.022586 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.022571    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:44.024443 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.024429    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:44.024503 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.024458    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:44.024503 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.024475    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:44.025016 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.025005    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 18:15:44.025016 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.025017    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 18:15:44.025149 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.025005    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:44.025149 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.025035    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:15:44.025722 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.025704    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:44.025770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.025731    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:44.025770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.025740    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:44.026762 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.026696    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-92.ec2.internal.18d456f0c24619fd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-92.ec2.internal,UID:ip-10-0-143-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-92.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-92.ec2.internal,},FirstTimestamp:2026-09-11 18:15:44.024443389 +0000 UTC m=+0.423242249,LastTimestamp:2026-09-11 18:15:44.024443389 +0000 UTC m=+0.423242249,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-92.ec2.internal,}"
Sep 11 18:15:44.027006 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.026994    2563 policy_none.go:49] "None policy: Start"
Sep 11 18:15:44.027078 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.027012    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 18:15:44.027078 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.027026    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 18:15:44.028991 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.028974    2563 policy_none.go:47] "Start"
Sep 11 18:15:44.086057 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.062996    2563 manager.go:337] "Starting Device Plugin manager"
Sep 11 18:15:44.086057 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.063024    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 18:15:44.086057 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.063033    2563 server.go:87] "Starting device plugin registration server"
Sep 11 18:15:44.086057 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.063374    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 18:15:44.086057 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.063386    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 18:15:44.086057 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.063469    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 18:15:44.086057 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.063561    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 18:15:44.086057 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.063571    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 18:15:44.086057 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.065292    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 18:15:44.086057 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.065351    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-92.ec2.internal\" not found"
Sep 11 18:15:44.100826 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.100795    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 18:15:44.101911 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.101885    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 18:15:44.101987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.101945    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 18:15:44.101987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.101971    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 18:15:44.102084 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.102013    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 18:15:44.104274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.104258    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:44.163803 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.163779    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:44.164526 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.164512    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:44.164637 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.164535    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:44.164637 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.164546    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:44.164637 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.164563    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.170517 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.170502    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.170575 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.170521    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-92.ec2.internal\": node \"ip-10-0-143-92.ec2.internal\" not found"
Sep 11 18:15:44.190157 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.190138    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-92.ec2.internal\" not found"
Sep 11 18:15:44.202712 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.202693    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-92.ec2.internal"]
Sep 11 18:15:44.202781 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.202752    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:44.203575 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.203557    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:44.203657 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.203585    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:44.203657 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.203595    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:44.205665 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.205654    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:44.205792 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.205780    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.205829 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.205805    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:44.206977 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.206960    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:44.207052 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.206983    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:44.207052 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.206992    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:44.207052 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.206969    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:44.207162 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.207055    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:44.207162 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.207066    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:44.209098 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.209070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.209167 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.209111    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:44.209745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.209729    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:44.209831 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.209758    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:44.209831 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.209772    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:44.224595 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.224579    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-92.ec2.internal\" not found" node="ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.228371 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.228356    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-92.ec2.internal\" not found" node="ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.290831 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.290811    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-92.ec2.internal\" not found"
Sep 11 18:15:44.307144 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.307121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e92ce411a910a52f68761a7f80b8212d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal\" (UID: \"e92ce411a910a52f68761a7f80b8212d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.307222 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.307147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e92ce411a910a52f68761a7f80b8212d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal\" (UID: \"e92ce411a910a52f68761a7f80b8212d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.307222 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.307163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d29890478aea68f0049d36cf8ba8661-config\") pod \"kube-apiserver-proxy-ip-10-0-143-92.ec2.internal\" (UID: \"3d29890478aea68f0049d36cf8ba8661\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.391934 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.391915    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-92.ec2.internal\" not found"
Sep 11 18:15:44.408230 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.408202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e92ce411a910a52f68761a7f80b8212d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal\" (UID: \"e92ce411a910a52f68761a7f80b8212d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.408313 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.408239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e92ce411a910a52f68761a7f80b8212d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal\" (UID: \"e92ce411a910a52f68761a7f80b8212d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.408313 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.408264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d29890478aea68f0049d36cf8ba8661-config\") pod \"kube-apiserver-proxy-ip-10-0-143-92.ec2.internal\" (UID: \"3d29890478aea68f0049d36cf8ba8661\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.408394 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.408314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e92ce411a910a52f68761a7f80b8212d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal\" (UID: \"e92ce411a910a52f68761a7f80b8212d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.408394 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.408330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e92ce411a910a52f68761a7f80b8212d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal\" (UID: \"e92ce411a910a52f68761a7f80b8212d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.408394 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.408316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d29890478aea68f0049d36cf8ba8661-config\") pod \"kube-apiserver-proxy-ip-10-0-143-92.ec2.internal\" (UID: \"3d29890478aea68f0049d36cf8ba8661\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.492635 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.492598    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-92.ec2.internal\" not found"
Sep 11 18:15:44.526111 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.526076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.530547 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:44.530531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:44.592987 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.592964    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-92.ec2.internal\" not found"
Sep 11 18:15:44.693519 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.693501    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-92.ec2.internal\" not found"
Sep 11 18:15:44.794047 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.794003    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-92.ec2.internal\" not found"
Sep 11 18:15:44.894477 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:44.894458    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-92.ec2.internal\" not found"
Sep 11 18:15:45.941271 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:45.941252    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 18:15:45.941596 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:45.941353    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 18:15:45.963014 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:45.962994    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:45.985100 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:45.985082    2563 apiserver.go:52] "Watching apiserver"
Sep 11 18:15:45.994617 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:45.994598    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 18:15:45.995618 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:45.995596    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-4gfk7","openshift-ovn-kubernetes/ovnkube-node-442rc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj","openshift-cluster-node-tuning-operator/tuned-489vf","openshift-dns/node-resolver-qfkcq","openshift-image-registry/node-ca-zkrtp","openshift-network-diagnostics/network-check-target-nw8l8","kube-system/global-pull-secret-syncer-phxl4","kube-system/konnectivity-agent-n8qwz","openshift-multus/multus-948kd","openshift-multus/multus-additional-cni-plugins-qlqkh","openshift-multus/network-metrics-daemon-tc5jj"]
Sep 11 18:15:45.998673 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:45.998655    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4gfk7"
Sep 11 18:15:46.001114 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.001098    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:15:46.001421 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.001405    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hqng8\""
Sep 11 18:15:46.001523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.001504    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 18:15:46.001735 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.001723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.001793 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.001734    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 18:15:46.003776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.003760    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 18:15:46.003904 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.003889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.004955 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.004930    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 18:15:46.005646 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.005192    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 18:15:46.005646 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.005452    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 18:15:46.005646 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.005506    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 18:15:46.005646 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.005582    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n7lhd\""
Sep 11 18:15:46.006061 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.005904    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 18:15:46.006821 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.006464    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 18:15:46.006821 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.006762    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zprh5\""
Sep 11 18:15:46.006971 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.006928    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 18:15:46.007024 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.006964    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 18:15:46.007278 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.007242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.009071 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.009056    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 18:15:46.009647 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.009631    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lkmwq\""
Sep 11 18:15:46.009743 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.009728    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:46.009977 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.009960    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:15:46.010054 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.009964    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 18:15:46.010163 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.010148    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qfkcq"
Sep 11 18:15:46.012260 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.012246    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 18:15:46.012447 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.012435    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 18:15:46.012522 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.012459    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qx7q7\""
Sep 11 18:15:46.012836 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.012823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zkrtp"
Sep 11 18:15:46.015021 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015006    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 18:15:46.015116 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015073    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x45gx\""
Sep 11 18:15:46.015116 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015105    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 18:15:46.015263 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df16d6da-3b2d-4191-97af-de284cef6b8a-host-slash\") pod \"iptables-alerter-4gfk7\" (UID: \"df16d6da-3b2d-4191-97af-de284cef6b8a\") " pod="openshift-network-operator/iptables-alerter-4gfk7"
Sep 11 18:15:46.015338 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015278    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 18:15:46.015338 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-var-lib-openvswitch\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.015338 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-kubernetes\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.015338 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-systemd\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.015582 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-run\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.015582 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-sys\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.015582 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-tuned\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.015582 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ptf\" (UniqueName: \"kubernetes.io/projected/4f5a15bf-eecf-420f-a95f-bc0153e33007-kube-api-access-56ptf\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.015582 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-kubelet\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.015582 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-node-log\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.015582 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-cni-bin\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-cni-netd\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f5a15bf-eecf-420f-a95f-bc0153e33007-tmp\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df16d6da-3b2d-4191-97af-de284cef6b8a-iptables-alerter-script\") pod \"iptables-alerter-4gfk7\" (UID: \"df16d6da-3b2d-4191-97af-de284cef6b8a\") " pod="openshift-network-operator/iptables-alerter-4gfk7"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-run-openvswitch\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8248d578-0ff4-4992-b9b0-3327f5d8e663-ovnkube-script-lib\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.015690    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5rj\" (UniqueName: \"kubernetes.io/projected/8248d578-0ff4-4992-b9b0-3327f5d8e663-kube-api-access-mg5rj\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5lc\" (UniqueName: \"kubernetes.io/projected/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-kube-api-access-gz5lc\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-modprobe-d\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-lib-modules\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.015848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-host\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lgkh\" (UniqueName: \"kubernetes.io/projected/df16d6da-3b2d-4191-97af-de284cef6b8a-kube-api-access-7lgkh\") pod \"iptables-alerter-4gfk7\" (UID: \"df16d6da-3b2d-4191-97af-de284cef6b8a\") " pod="openshift-network-operator/iptables-alerter-4gfk7"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3725edf8-bf10-4f95-99cb-2dc2581932a4-hosts-file\") pod \"node-resolver-qfkcq\" (UID: \"3725edf8-bf10-4f95-99cb-2dc2581932a4\") " pod="openshift-dns/node-resolver-qfkcq"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0-serviceca\") pod \"node-ca-zkrtp\" (UID: \"9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0\") " pod="openshift-image-registry/node-ca-zkrtp"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfx9w\" (UniqueName: \"kubernetes.io/projected/9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0-kube-api-access-sfx9w\") pod \"node-ca-zkrtp\" (UID: \"9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0\") " pod="openshift-image-registry/node-ca-zkrtp"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-slash\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.015978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-run-netns\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mnb\" (UniqueName: \"kubernetes.io/projected/3725edf8-bf10-4f95-99cb-2dc2581932a4-kube-api-access-v7mnb\") pod \"node-resolver-qfkcq\" (UID: \"3725edf8-bf10-4f95-99cb-2dc2581932a4\") " pod="openshift-dns/node-resolver-qfkcq"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-systemd-units\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-run-systemd\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-run-ovn\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-log-socket\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8248d578-0ff4-4992-b9b0-3327f5d8e663-env-overrides\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8248d578-0ff4-4992-b9b0-3327f5d8e663-ovn-node-metrics-cert\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0-host\") pod \"node-ca-zkrtp\" (UID: \"9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0\") " pod="openshift-image-registry/node-ca-zkrtp"
Sep 11 18:15:46.016471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-etc-openvswitch\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-run-ovn-kubernetes\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-registration-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-sysconfig\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-sysctl-d\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-sysctl-conf\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3725edf8-bf10-4f95-99cb-2dc2581932a4-tmp-dir\") pod \"node-resolver-qfkcq\" (UID: \"3725edf8-bf10-4f95-99cb-2dc2581932a4\") " pod="openshift-dns/node-resolver-qfkcq"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8248d578-0ff4-4992-b9b0-3327f5d8e663-ovnkube-config\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-socket-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-device-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-sys-fs\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.017011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.016548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-var-lib-kubelet\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.018857 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.018838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:46.018955 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.018934    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:15:46.019017 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.018979    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 18:10:45 +0000 UTC" deadline="2028-04-04 16:32:04.81832149 +0000 UTC"
Sep 11 18:15:46.019063 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.019020    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13702h16m18.799305103s"
Sep 11 18:15:46.020795 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.020694    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:15:46.020795 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.020767    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-92.ec2.internal"
Sep 11 18:15:46.022044 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.022027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:15:46.024657 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.024626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.025412 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.025391    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 18:15:46.025550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.025427    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-49gw6\""
Sep 11 18:15:46.025550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.025455    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 18:15:46.026836 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.026791    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 18:15:46.026836 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.026821    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 18:15:46.027145 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.027130    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 18:15:46.027247 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.027167    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 18:15:46.027303 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.027284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.027647 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.027631    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9m9gh\""
Sep 11 18:15:46.027647 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.027645    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:15:46.029648 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.029632    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 18:15:46.029736 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.029688    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zddnf\""
Sep 11 18:15:46.029814 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.029799    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 18:15:46.029814 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.029799    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal"]
Sep 11 18:15:46.029910 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.029894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:46.029967 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.029950    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:15:46.031941 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.031927    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:15:46.032128 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.032114    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-92.ec2.internal"]
Sep 11 18:15:46.049296 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.049254    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qvc5w"
Sep 11 18:15:46.056085 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.055950    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qvc5w"
Sep 11 18:15:46.111429 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.111411    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 18:15:46.117456 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3725edf8-bf10-4f95-99cb-2dc2581932a4-tmp-dir\") pod \"node-resolver-qfkcq\" (UID: \"3725edf8-bf10-4f95-99cb-2dc2581932a4\") " pod="openshift-dns/node-resolver-qfkcq"
Sep 11 18:15:46.117529 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8248d578-0ff4-4992-b9b0-3327f5d8e663-ovnkube-config\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.117529 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-device-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.117529 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-var-lib-kubelet\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.117529 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df16d6da-3b2d-4191-97af-de284cef6b8a-host-slash\") pod \"iptables-alerter-4gfk7\" (UID: \"df16d6da-3b2d-4191-97af-de284cef6b8a\") " pod="openshift-network-operator/iptables-alerter-4gfk7"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-kubernetes\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-run\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-device-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-var-lib-kubelet\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-sys\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df16d6da-3b2d-4191-97af-de284cef6b8a-host-slash\") pod \"iptables-alerter-4gfk7\" (UID: \"df16d6da-3b2d-4191-97af-de284cef6b8a\") " pod="openshift-network-operator/iptables-alerter-4gfk7"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc5cc477-94d4-4770-8901-f1e8813f8ebc-agent-certs\") pod \"konnectivity-agent-n8qwz\" (UID: \"cc5cc477-94d4-4770-8901-f1e8813f8ebc\") " pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-run-netns\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-run\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-kubernetes\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-etc-kubernetes\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbj65\" (UniqueName: \"kubernetes.io/projected/13979e2d-de4f-4ee8-84ac-6595911215cc-kube-api-access-nbj65\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.117776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3725edf8-bf10-4f95-99cb-2dc2581932a4-tmp-dir\") pod \"node-resolver-qfkcq\" (UID: \"3725edf8-bf10-4f95-99cb-2dc2581932a4\") " pod="openshift-dns/node-resolver-qfkcq"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-sys\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-kubelet\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-kubelet\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-socket-dir-parent\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-var-lib-cni-multus\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-cni-netd\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsb75\" (UniqueName: \"kubernetes.io/projected/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-kube-api-access-zsb75\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-hostroot\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8248d578-0ff4-4992-b9b0-3327f5d8e663-ovnkube-config\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-run-multus-certs\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-cni-netd\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.117998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13979e2d-de4f-4ee8-84ac-6595911215cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-run-openvswitch\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8248d578-0ff4-4992-b9b0-3327f5d8e663-ovnkube-script-lib\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5rj\" (UniqueName: \"kubernetes.io/projected/8248d578-0ff4-4992-b9b0-3327f5d8e663-kube-api-access-mg5rj\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-lib-modules\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.118284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118092    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-run-openvswitch\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-sys-fs\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3725edf8-bf10-4f95-99cb-2dc2581932a4-hosts-file\") pod \"node-resolver-qfkcq\" (UID: \"3725edf8-bf10-4f95-99cb-2dc2581932a4\") " pod="openshift-dns/node-resolver-qfkcq"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0-serviceca\") pod \"node-ca-zkrtp\" (UID: \"9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0\") " pod="openshift-image-registry/node-ca-zkrtp"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfx9w\" (UniqueName: \"kubernetes.io/projected/9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0-kube-api-access-sfx9w\") pod \"node-ca-zkrtp\" (UID: \"9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0\") " pod="openshift-image-registry/node-ca-zkrtp"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-lib-modules\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8248d578-0ff4-4992-b9b0-3327f5d8e663-ovn-node-metrics-cert\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-sys-fs\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3725edf8-bf10-4f95-99cb-2dc2581932a4-hosts-file\") pod \"node-resolver-qfkcq\" (UID: \"3725edf8-bf10-4f95-99cb-2dc2581932a4\") " pod="openshift-dns/node-resolver-qfkcq"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-cnibin\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ptf\" (UniqueName: \"kubernetes.io/projected/4f5a15bf-eecf-420f-a95f-bc0153e33007-kube-api-access-56ptf\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mnb\" (UniqueName: \"kubernetes.io/projected/3725edf8-bf10-4f95-99cb-2dc2581932a4-kube-api-access-v7mnb\") pod \"node-resolver-qfkcq\" (UID: \"3725edf8-bf10-4f95-99cb-2dc2581932a4\") " pod="openshift-dns/node-resolver-qfkcq"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8248d578-0ff4-4992-b9b0-3327f5d8e663-env-overrides\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-sysconfig\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.119171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc5cc477-94d4-4770-8901-f1e8813f8ebc-konnectivity-ca\") pod \"konnectivity-agent-n8qwz\" (UID: \"cc5cc477-94d4-4770-8901-f1e8813f8ebc\") " pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-system-cni-dir\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee8197a9-86c5-472a-8d94-d7b0de84b73f-cni-binary-copy\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0-host\") pod \"node-ca-zkrtp\" (UID: \"9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0\") " pod="openshift-image-registry/node-ca-zkrtp"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8248d578-0ff4-4992-b9b0-3327f5d8e663-ovnkube-script-lib\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-etc-openvswitch\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0-host\") pod \"node-ca-zkrtp\" (UID: \"9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0\") " pod="openshift-image-registry/node-ca-zkrtp"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-sysconfig\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-registration-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-etc-openvswitch\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-sysctl-d\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-registration-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0-serviceca\") pod \"node-ca-zkrtp\" (UID: \"9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0\") " pod="openshift-image-registry/node-ca-zkrtp"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-sysctl-conf\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.120001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-sysctl-d\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118760    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-var-lib-cni-bin\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-sysctl-conf\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9fhw\" (UniqueName: \"kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw\") pod \"network-check-target-nw8l8\" (UID: \"0b26c0e1-d27d-4cb5-9731-8a321f327115\") " pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-var-lib-kubelet\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvmcp\" (UniqueName: \"kubernetes.io/projected/ee8197a9-86c5-472a-8d94-d7b0de84b73f-kube-api-access-zvmcp\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13979e2d-de4f-4ee8-84ac-6595911215cc-cnibin\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8248d578-0ff4-4992-b9b0-3327f5d8e663-env-overrides\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.118986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-var-lib-openvswitch\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-systemd\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-cni-dir\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-var-lib-openvswitch\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13979e2d-de4f-4ee8-84ac-6595911215cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-systemd\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.120528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-node-log\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-cni-bin\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-node-log\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13979e2d-de4f-4ee8-84ac-6595911215cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-cni-bin\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f5a15bf-eecf-420f-a95f-bc0153e33007-tmp\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-kubelet-config\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-daemon-config\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13979e2d-de4f-4ee8-84ac-6595911215cc-system-cni-dir\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df16d6da-3b2d-4191-97af-de284cef6b8a-iptables-alerter-script\") pod \"iptables-alerter-4gfk7\" (UID: \"df16d6da-3b2d-4191-97af-de284cef6b8a\") " pod="openshift-network-operator/iptables-alerter-4gfk7"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5lc\" (UniqueName: \"kubernetes.io/projected/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-kube-api-access-gz5lc\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-modprobe-d\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-host\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-socket-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lgkh\" (UniqueName: \"kubernetes.io/projected/df16d6da-3b2d-4191-97af-de284cef6b8a-kube-api-access-7lgkh\") pod \"iptables-alerter-4gfk7\" (UID: \"df16d6da-3b2d-4191-97af-de284cef6b8a\") " pod="openshift-network-operator/iptables-alerter-4gfk7"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-slash\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-modprobe-d\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.121030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-run-netns\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-run-netns\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-socket-dir\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-slash\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-run-systemd\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-run-systemd\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f5a15bf-eecf-420f-a95f-bc0153e33007-host\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-dbus\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-tuned\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-systemd-units\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-run-ovn\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-log-socket\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-systemd-units\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-run-ovn\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-run-k8s-cni-cncf-io\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-log-socket\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df16d6da-3b2d-4191-97af-de284cef6b8a-iptables-alerter-script\") pod \"iptables-alerter-4gfk7\" (UID: \"df16d6da-3b2d-4191-97af-de284cef6b8a\") " pod="openshift-network-operator/iptables-alerter-4gfk7"
Sep 11 18:15:46.121523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13979e2d-de4f-4ee8-84ac-6595911215cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.122276 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-run-ovn-kubernetes\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.122276 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-conf-dir\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.122276 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.119971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8248d578-0ff4-4992-b9b0-3327f5d8e663-host-run-ovn-kubernetes\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.122533 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.122310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8248d578-0ff4-4992-b9b0-3327f5d8e663-ovn-node-metrics-cert\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.122533 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.122350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f5a15bf-eecf-420f-a95f-bc0153e33007-etc-tuned\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.122533 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.122392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f5a15bf-eecf-420f-a95f-bc0153e33007-tmp\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.129556 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.129536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfx9w\" (UniqueName: \"kubernetes.io/projected/9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0-kube-api-access-sfx9w\") pod \"node-ca-zkrtp\" (UID: \"9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0\") " pod="openshift-image-registry/node-ca-zkrtp"
Sep 11 18:15:46.131683 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.131663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5lc\" (UniqueName: \"kubernetes.io/projected/b0d38a1d-e3f2-45d9-bdce-fb3530732ed0-kube-api-access-gz5lc\") pod \"aws-ebs-csi-driver-node-gtqjj\" (UID: \"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.132142 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.132117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mnb\" (UniqueName: \"kubernetes.io/projected/3725edf8-bf10-4f95-99cb-2dc2581932a4-kube-api-access-v7mnb\") pod \"node-resolver-qfkcq\" (UID: \"3725edf8-bf10-4f95-99cb-2dc2581932a4\") " pod="openshift-dns/node-resolver-qfkcq"
Sep 11 18:15:46.132231 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.132199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lgkh\" (UniqueName: \"kubernetes.io/projected/df16d6da-3b2d-4191-97af-de284cef6b8a-kube-api-access-7lgkh\") pod \"iptables-alerter-4gfk7\" (UID: \"df16d6da-3b2d-4191-97af-de284cef6b8a\") " pod="openshift-network-operator/iptables-alerter-4gfk7"
Sep 11 18:15:46.132365 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.132347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ptf\" (UniqueName: \"kubernetes.io/projected/4f5a15bf-eecf-420f-a95f-bc0153e33007-kube-api-access-56ptf\") pod \"tuned-489vf\" (UID: \"4f5a15bf-eecf-420f-a95f-bc0153e33007\") " pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.132667 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.132634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5rj\" (UniqueName: \"kubernetes.io/projected/8248d578-0ff4-4992-b9b0-3327f5d8e663-kube-api-access-mg5rj\") pod \"ovnkube-node-442rc\" (UID: \"8248d578-0ff4-4992-b9b0-3327f5d8e663\") " pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.173047 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:46.173020    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d29890478aea68f0049d36cf8ba8661.slice/crio-93931d6538c635122b2592bb5b417dab47fad91e0de5cbf7230339677568f178 WatchSource:0}: Error finding container 93931d6538c635122b2592bb5b417dab47fad91e0de5cbf7230339677568f178: Status 404 returned error can't find the container with id 93931d6538c635122b2592bb5b417dab47fad91e0de5cbf7230339677568f178
Sep 11 18:15:46.173234 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:46.173216    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92ce411a910a52f68761a7f80b8212d.slice/crio-8c7fe3b546c78885d73059175304643bb5fbaa8eba0512cda8bae656913d4a4c WatchSource:0}: Error finding container 8c7fe3b546c78885d73059175304643bb5fbaa8eba0512cda8bae656913d4a4c: Status 404 returned error can't find the container with id 8c7fe3b546c78885d73059175304643bb5fbaa8eba0512cda8bae656913d4a4c
Sep 11 18:15:46.178944 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.178931    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:15:46.220680 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-var-lib-cni-bin\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.220765 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fhw\" (UniqueName: \"kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw\") pod \"network-check-target-nw8l8\" (UID: \"0b26c0e1-d27d-4cb5-9731-8a321f327115\") " pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:46.220765 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:46.220765 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-var-lib-kubelet\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-var-lib-cni-bin\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvmcp\" (UniqueName: \"kubernetes.io/projected/ee8197a9-86c5-472a-8d94-d7b0de84b73f-kube-api-access-zvmcp\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.220776    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13979e2d-de4f-4ee8-84ac-6595911215cc-cnibin\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-cni-dir\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13979e2d-de4f-4ee8-84ac-6595911215cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-var-lib-kubelet\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.220877    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret podName:693fe3c2-fe4c-4e69-9922-8b23c60e5d5e nodeName:}" failed. No retries permitted until 2026-09-11 18:15:46.72083635 +0000 UTC m=+2.115351361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret") pod "global-pull-secret-syncer-phxl4" (UID: "693fe3c2-fe4c-4e69-9922-8b23c60e5d5e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13979e2d-de4f-4ee8-84ac-6595911215cc-cnibin\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13979e2d-de4f-4ee8-84ac-6595911215cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-kubelet-config\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:46.220956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-daemon-config\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13979e2d-de4f-4ee8-84ac-6595911215cc-system-cni-dir\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-dbus\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13979e2d-de4f-4ee8-84ac-6595911215cc-system-cni-dir\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-kubelet-config\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-run-k8s-cni-cncf-io\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.220967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-cni-dir\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13979e2d-de4f-4ee8-84ac-6595911215cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221073    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-dbus\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-run-k8s-cni-cncf-io\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-conf-dir\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc5cc477-94d4-4770-8901-f1e8813f8ebc-agent-certs\") pod \"konnectivity-agent-n8qwz\" (UID: \"cc5cc477-94d4-4770-8901-f1e8813f8ebc\") " pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-run-netns\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-etc-kubernetes\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbj65\" (UniqueName: \"kubernetes.io/projected/13979e2d-de4f-4ee8-84ac-6595911215cc-kube-api-access-nbj65\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-run-netns\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-conf-dir\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.221550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13979e2d-de4f-4ee8-84ac-6595911215cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-socket-dir-parent\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-etc-kubernetes\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-socket-dir-parent\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-var-lib-cni-multus\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsb75\" (UniqueName: \"kubernetes.io/projected/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-kube-api-access-zsb75\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-hostroot\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13979e2d-de4f-4ee8-84ac-6595911215cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-run-multus-certs\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13979e2d-de4f-4ee8-84ac-6595911215cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-var-lib-cni-multus\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-cnibin\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-host-run-multus-certs\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc5cc477-94d4-4770-8901-f1e8813f8ebc-konnectivity-ca\") pod \"konnectivity-agent-n8qwz\" (UID: \"cc5cc477-94d4-4770-8901-f1e8813f8ebc\") " pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-hostroot\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-system-cni-dir\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee8197a9-86c5-472a-8d94-d7b0de84b73f-cni-binary-copy\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:46.222112 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13979e2d-de4f-4ee8-84ac-6595911215cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.222639 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee8197a9-86c5-472a-8d94-d7b0de84b73f-multus-daemon-config\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222639 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-cnibin\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222639 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.221669    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:46.222639 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.221722    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs podName:d24dd9d4-b46c-4d50-8d1e-7b494d53cb17 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:46.721707085 +0000 UTC m=+2.116222103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs") pod "network-metrics-daemon-tc5jj" (UID: "d24dd9d4-b46c-4d50-8d1e-7b494d53cb17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:46.222639 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221735    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee8197a9-86c5-472a-8d94-d7b0de84b73f-system-cni-dir\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.222639 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13979e2d-de4f-4ee8-84ac-6595911215cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.222639 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc5cc477-94d4-4770-8901-f1e8813f8ebc-konnectivity-ca\") pod \"konnectivity-agent-n8qwz\" (UID: \"cc5cc477-94d4-4770-8901-f1e8813f8ebc\") " pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:15:46.222639 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.221969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee8197a9-86c5-472a-8d94-d7b0de84b73f-cni-binary-copy\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.223146 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.223130    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc5cc477-94d4-4770-8901-f1e8813f8ebc-agent-certs\") pod \"konnectivity-agent-n8qwz\" (UID: \"cc5cc477-94d4-4770-8901-f1e8813f8ebc\") " pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:15:46.230480 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.230461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvmcp\" (UniqueName: \"kubernetes.io/projected/ee8197a9-86c5-472a-8d94-d7b0de84b73f-kube-api-access-zvmcp\") pod \"multus-948kd\" (UID: \"ee8197a9-86c5-472a-8d94-d7b0de84b73f\") " pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.230729 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.230714    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:46.230729 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.230730    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:46.230844 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.230739    2563 projected.go:196] Error preparing data for projected volume kube-api-access-h9fhw for pod openshift-network-diagnostics/network-check-target-nw8l8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:46.230844 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.230786    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw podName:0b26c0e1-d27d-4cb5-9731-8a321f327115 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:46.730774652 +0000 UTC m=+2.125289680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h9fhw" (UniqueName: "kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw") pod "network-check-target-nw8l8" (UID: "0b26c0e1-d27d-4cb5-9731-8a321f327115") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:46.232005 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.231990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbj65\" (UniqueName: \"kubernetes.io/projected/13979e2d-de4f-4ee8-84ac-6595911215cc-kube-api-access-nbj65\") pod \"multus-additional-cni-plugins-qlqkh\" (UID: \"13979e2d-de4f-4ee8-84ac-6595911215cc\") " pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.232419 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.232363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsb75\" (UniqueName: \"kubernetes.io/projected/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-kube-api-access-zsb75\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:46.326825 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.326782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4gfk7"
Sep 11 18:15:46.335123 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:46.335107    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf16d6da_3b2d_4191_97af_de284cef6b8a.slice/crio-1294c381ff2c7b6c427eebb52e64bce73ee1e04632f345904c96fc65287c9aef WatchSource:0}: Error finding container 1294c381ff2c7b6c427eebb52e64bce73ee1e04632f345904c96fc65287c9aef: Status 404 returned error can't find the container with id 1294c381ff2c7b6c427eebb52e64bce73ee1e04632f345904c96fc65287c9aef
Sep 11 18:15:46.340043 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.340028    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:15:46.347135 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:46.347118    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8248d578_0ff4_4992_b9b0_3327f5d8e663.slice/crio-fd59ce7fdc44c6cec88cf2a2ac2962ce1a940579a724fc4bd30b1fd8c2de2775 WatchSource:0}: Error finding container fd59ce7fdc44c6cec88cf2a2ac2962ce1a940579a724fc4bd30b1fd8c2de2775: Status 404 returned error can't find the container with id fd59ce7fdc44c6cec88cf2a2ac2962ce1a940579a724fc4bd30b1fd8c2de2775
Sep 11 18:15:46.351022 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.351006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj"
Sep 11 18:15:46.358078 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:46.358055    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d38a1d_e3f2_45d9_bdce_fb3530732ed0.slice/crio-64a603e4858be35685a080334b0d4b0b6dbaf9c62505c4775f16c1bcd21ad396 WatchSource:0}: Error finding container 64a603e4858be35685a080334b0d4b0b6dbaf9c62505c4775f16c1bcd21ad396: Status 404 returned error can't find the container with id 64a603e4858be35685a080334b0d4b0b6dbaf9c62505c4775f16c1bcd21ad396
Sep 11 18:15:46.363057 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.363043    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-489vf"
Sep 11 18:15:46.370289 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:46.370267    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5a15bf_eecf_420f_a95f_bc0153e33007.slice/crio-a8ad60c0f324938cc06dd864f9a0c13cd967b73d1dfe61d21e2ac0a442530146 WatchSource:0}: Error finding container a8ad60c0f324938cc06dd864f9a0c13cd967b73d1dfe61d21e2ac0a442530146: Status 404 returned error can't find the container with id a8ad60c0f324938cc06dd864f9a0c13cd967b73d1dfe61d21e2ac0a442530146
Sep 11 18:15:46.370727 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.370710    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:46.385513 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.385496    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qfkcq"
Sep 11 18:15:46.392108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.392092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zkrtp"
Sep 11 18:15:46.392612 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:46.392595    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3725edf8_bf10_4f95_99cb_2dc2581932a4.slice/crio-710ed9c80179e713d690110fdb25cc9829b25870b8f52f7abdbb0f90b81c146d WatchSource:0}: Error finding container 710ed9c80179e713d690110fdb25cc9829b25870b8f52f7abdbb0f90b81c146d: Status 404 returned error can't find the container with id 710ed9c80179e713d690110fdb25cc9829b25870b8f52f7abdbb0f90b81c146d
Sep 11 18:15:46.399892 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:46.399873    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b80a3a2_8a1b_4382_a49f_ac7a667d5aa0.slice/crio-8200f4c2778fe8862e9f85c49e0998a7492e1025344fff3ec99c04e5df5b5a57 WatchSource:0}: Error finding container 8200f4c2778fe8862e9f85c49e0998a7492e1025344fff3ec99c04e5df5b5a57: Status 404 returned error can't find the container with id 8200f4c2778fe8862e9f85c49e0998a7492e1025344fff3ec99c04e5df5b5a57
Sep 11 18:15:46.400044 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.400030    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:46.407964 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.407947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:15:46.414944 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:46.414928    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5cc477_94d4_4770_8901_f1e8813f8ebc.slice/crio-0c483f712e7db83d790eca424a36c949e0dad129f288a44b95206be492429327 WatchSource:0}: Error finding container 0c483f712e7db83d790eca424a36c949e0dad129f288a44b95206be492429327: Status 404 returned error can't find the container with id 0c483f712e7db83d790eca424a36c949e0dad129f288a44b95206be492429327
Sep 11 18:15:46.423992 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.423978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-948kd"
Sep 11 18:15:46.431132 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:46.431116    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8197a9_86c5_472a_8d94_d7b0de84b73f.slice/crio-3529d8a5da518d95980c44c81ac878e285868cd6b68314c71a141c9ee974d175 WatchSource:0}: Error finding container 3529d8a5da518d95980c44c81ac878e285868cd6b68314c71a141c9ee974d175: Status 404 returned error can't find the container with id 3529d8a5da518d95980c44c81ac878e285868cd6b68314c71a141c9ee974d175
Sep 11 18:15:46.439946 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.439925    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qlqkh"
Sep 11 18:15:46.449762 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:15:46.449743    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13979e2d_de4f_4ee8_84ac_6595911215cc.slice/crio-149f20c127b5f301edb643b19f7db46cdd6d4b12cf47240d30503329e7cbc418 WatchSource:0}: Error finding container 149f20c127b5f301edb643b19f7db46cdd6d4b12cf47240d30503329e7cbc418: Status 404 returned error can't find the container with id 149f20c127b5f301edb643b19f7db46cdd6d4b12cf47240d30503329e7cbc418
Sep 11 18:15:46.724680 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.724638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:46.724835 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.724735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:46.724887 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.724872    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:46.724941 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.724930    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret podName:693fe3c2-fe4c-4e69-9922-8b23c60e5d5e nodeName:}" failed. No retries permitted until 2026-09-11 18:15:47.724913731 +0000 UTC m=+3.119428739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret") pod "global-pull-secret-syncer-phxl4" (UID: "693fe3c2-fe4c-4e69-9922-8b23c60e5d5e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:46.725003 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.724982    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:46.725094 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.725066    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs podName:d24dd9d4-b46c-4d50-8d1e-7b494d53cb17 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:47.725049516 +0000 UTC m=+3.119564531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs") pod "network-metrics-daemon-tc5jj" (UID: "d24dd9d4-b46c-4d50-8d1e-7b494d53cb17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:46.754033 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.754002    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:46.825865 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:46.825833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fhw\" (UniqueName: \"kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw\") pod \"network-check-target-nw8l8\" (UID: \"0b26c0e1-d27d-4cb5-9731-8a321f327115\") " pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:46.826003 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.825995    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:46.826072 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.826015    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:46.826072 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.826028    2563 projected.go:196] Error preparing data for projected volume kube-api-access-h9fhw for pod openshift-network-diagnostics/network-check-target-nw8l8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:46.826179 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:46.826079    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw podName:0b26c0e1-d27d-4cb5-9731-8a321f327115 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:47.826062333 +0000 UTC m=+3.220577361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h9fhw" (UniqueName: "kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw") pod "network-check-target-nw8l8" (UID: "0b26c0e1-d27d-4cb5-9731-8a321f327115") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:47.062507 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.062336    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:46 +0000 UTC" deadline="2028-02-16 00:34:37.986664095 +0000 UTC"
Sep 11 18:15:47.062507 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.062415    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12534h18m50.924253045s"
Sep 11 18:15:47.109037 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.109009    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:47.109184 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:47.109134    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:15:47.109537 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.109512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:47.109627 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:47.109604    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:15:47.133732 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.133677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4gfk7" event={"ID":"df16d6da-3b2d-4191-97af-de284cef6b8a","Type":"ContainerStarted","Data":"1294c381ff2c7b6c427eebb52e64bce73ee1e04632f345904c96fc65287c9aef"}
Sep 11 18:15:47.151703 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.151674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-92.ec2.internal" event={"ID":"3d29890478aea68f0049d36cf8ba8661","Type":"ContainerStarted","Data":"93931d6538c635122b2592bb5b417dab47fad91e0de5cbf7230339677568f178"}
Sep 11 18:15:47.174081 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.174057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qfkcq" event={"ID":"3725edf8-bf10-4f95-99cb-2dc2581932a4","Type":"ContainerStarted","Data":"710ed9c80179e713d690110fdb25cc9829b25870b8f52f7abdbb0f90b81c146d"}
Sep 11 18:15:47.190011 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.189973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj" event={"ID":"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0","Type":"ContainerStarted","Data":"64a603e4858be35685a080334b0d4b0b6dbaf9c62505c4775f16c1bcd21ad396"}
Sep 11 18:15:47.207891 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.207863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal" event={"ID":"e92ce411a910a52f68761a7f80b8212d","Type":"ContainerStarted","Data":"8c7fe3b546c78885d73059175304643bb5fbaa8eba0512cda8bae656913d4a4c"}
Sep 11 18:15:47.233947 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.233920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlqkh" event={"ID":"13979e2d-de4f-4ee8-84ac-6595911215cc","Type":"ContainerStarted","Data":"149f20c127b5f301edb643b19f7db46cdd6d4b12cf47240d30503329e7cbc418"}
Sep 11 18:15:47.246651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.246624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-948kd" event={"ID":"ee8197a9-86c5-472a-8d94-d7b0de84b73f","Type":"ContainerStarted","Data":"3529d8a5da518d95980c44c81ac878e285868cd6b68314c71a141c9ee974d175"}
Sep 11 18:15:47.259584 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.259560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n8qwz" event={"ID":"cc5cc477-94d4-4770-8901-f1e8813f8ebc","Type":"ContainerStarted","Data":"0c483f712e7db83d790eca424a36c949e0dad129f288a44b95206be492429327"}
Sep 11 18:15:47.292080 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.292056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zkrtp" event={"ID":"9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0","Type":"ContainerStarted","Data":"8200f4c2778fe8862e9f85c49e0998a7492e1025344fff3ec99c04e5df5b5a57"}
Sep 11 18:15:47.306173 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.306145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-489vf" event={"ID":"4f5a15bf-eecf-420f-a95f-bc0153e33007","Type":"ContainerStarted","Data":"a8ad60c0f324938cc06dd864f9a0c13cd967b73d1dfe61d21e2ac0a442530146"}
Sep 11 18:15:47.325599 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.324092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-442rc" event={"ID":"8248d578-0ff4-4992-b9b0-3327f5d8e663","Type":"ContainerStarted","Data":"fd59ce7fdc44c6cec88cf2a2ac2962ce1a940579a724fc4bd30b1fd8c2de2775"}
Sep 11 18:15:47.733715 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.733680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:47.733877 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.733741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:47.733877 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:47.733849    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:47.733967 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:47.733907    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret podName:693fe3c2-fe4c-4e69-9922-8b23c60e5d5e nodeName:}" failed. No retries permitted until 2026-09-11 18:15:49.733889562 +0000 UTC m=+5.128404579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret") pod "global-pull-secret-syncer-phxl4" (UID: "693fe3c2-fe4c-4e69-9922-8b23c60e5d5e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:47.735505 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:47.734326    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:47.735505 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:47.734402    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs podName:d24dd9d4-b46c-4d50-8d1e-7b494d53cb17 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:49.734385392 +0000 UTC m=+5.128900395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs") pod "network-metrics-daemon-tc5jj" (UID: "d24dd9d4-b46c-4d50-8d1e-7b494d53cb17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:47.834548 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:47.834516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fhw\" (UniqueName: \"kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw\") pod \"network-check-target-nw8l8\" (UID: \"0b26c0e1-d27d-4cb5-9731-8a321f327115\") " pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:47.834759 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:47.834741    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:47.834821 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:47.834770    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:47.834821 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:47.834782    2563 projected.go:196] Error preparing data for projected volume kube-api-access-h9fhw for pod openshift-network-diagnostics/network-check-target-nw8l8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:47.834927 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:47.834838    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw podName:0b26c0e1-d27d-4cb5-9731-8a321f327115 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:49.834820894 +0000 UTC m=+5.229335898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h9fhw" (UniqueName: "kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw") pod "network-check-target-nw8l8" (UID: "0b26c0e1-d27d-4cb5-9731-8a321f327115") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:48.062659 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:48.062544    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:46 +0000 UTC" deadline="2028-03-22 08:25:18.464740137 +0000 UTC"
Sep 11 18:15:48.062659 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:48.062612    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13382h9m30.402131992s"
Sep 11 18:15:48.107744 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:48.106527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:48.107744 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:48.106675    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:15:49.109782 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:49.109149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:49.109782 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:49.109267    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:15:49.109782 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:49.109648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:49.109782 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:49.109739    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:15:49.753753 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:49.753718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:49.753936 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:49.753785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:49.753936 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:49.753914    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:49.754055 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:49.753975    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret podName:693fe3c2-fe4c-4e69-9922-8b23c60e5d5e nodeName:}" failed. No retries permitted until 2026-09-11 18:15:53.753957042 +0000 UTC m=+9.148472062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret") pod "global-pull-secret-syncer-phxl4" (UID: "693fe3c2-fe4c-4e69-9922-8b23c60e5d5e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:49.754487 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:49.754359    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:49.754487 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:49.754426    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs podName:d24dd9d4-b46c-4d50-8d1e-7b494d53cb17 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:53.754410118 +0000 UTC m=+9.148925147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs") pod "network-metrics-daemon-tc5jj" (UID: "d24dd9d4-b46c-4d50-8d1e-7b494d53cb17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:49.854738 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:49.854090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fhw\" (UniqueName: \"kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw\") pod \"network-check-target-nw8l8\" (UID: \"0b26c0e1-d27d-4cb5-9731-8a321f327115\") " pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:49.854738 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:49.854276    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:49.854738 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:49.854298    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:49.854738 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:49.854312    2563 projected.go:196] Error preparing data for projected volume kube-api-access-h9fhw for pod openshift-network-diagnostics/network-check-target-nw8l8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:49.854738 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:49.854389    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw podName:0b26c0e1-d27d-4cb5-9731-8a321f327115 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:53.854351122 +0000 UTC m=+9.248866140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h9fhw" (UniqueName: "kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw") pod "network-check-target-nw8l8" (UID: "0b26c0e1-d27d-4cb5-9731-8a321f327115") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:50.107382 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:50.107291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:50.107549 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:50.107435    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:15:51.107390 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:51.106708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:51.107390 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:51.106844    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:15:51.107390 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:51.107215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:51.107390 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:51.107320    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:15:52.107389 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:52.107332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:52.107566 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:52.107499    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:15:53.107161 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:53.107111    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:53.107341 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:53.107111    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:53.107341 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:53.107246    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:15:53.107341 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:53.107314    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:15:53.786052 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:53.785552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:53.786052 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:53.785640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:53.786052 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:53.785700    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:53.786052 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:53.785710    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:53.786052 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:53.785759    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret podName:693fe3c2-fe4c-4e69-9922-8b23c60e5d5e nodeName:}" failed. No retries permitted until 2026-09-11 18:16:01.785741019 +0000 UTC m=+17.180256026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret") pod "global-pull-secret-syncer-phxl4" (UID: "693fe3c2-fe4c-4e69-9922-8b23c60e5d5e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:53.786052 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:53.785776    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs podName:d24dd9d4-b46c-4d50-8d1e-7b494d53cb17 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:01.785766346 +0000 UTC m=+17.180281356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs") pod "network-metrics-daemon-tc5jj" (UID: "d24dd9d4-b46c-4d50-8d1e-7b494d53cb17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:53.886468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:53.886383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fhw\" (UniqueName: \"kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw\") pod \"network-check-target-nw8l8\" (UID: \"0b26c0e1-d27d-4cb5-9731-8a321f327115\") " pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:53.886614 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:53.886557    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:53.886614 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:53.886582    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:53.886614 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:53.886594    2563 projected.go:196] Error preparing data for projected volume kube-api-access-h9fhw for pod openshift-network-diagnostics/network-check-target-nw8l8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:53.886757 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:53.886646    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw podName:0b26c0e1-d27d-4cb5-9731-8a321f327115 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:01.886629192 +0000 UTC m=+17.281144199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h9fhw" (UniqueName: "kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw") pod "network-check-target-nw8l8" (UID: "0b26c0e1-d27d-4cb5-9731-8a321f327115") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:54.107569 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:54.107058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:54.107569 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:54.107197    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:15:55.107430 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:55.107393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:55.107890 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:55.107453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:55.107890 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:55.107534    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:15:55.108001 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:55.107962    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:15:56.106967 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:56.106937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:56.107117 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:56.107045    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:15:57.106689 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:57.106656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:57.107220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:57.106656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:57.107220 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:57.106789    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:15:57.107220 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:57.106839    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:15:58.106673 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:58.106639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:15:58.106837 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:58.106759    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:15:59.107213 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:59.107176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:15:59.107657 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:15:59.107228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:15:59.107657 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:59.107305    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:15:59.107657 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:15:59.107473    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:16:00.106801 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:00.106765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:00.106978 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:00.106885    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:16:01.106929 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:01.106890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:01.107331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:01.106890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:01.107331 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:01.107005    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:16:01.107331 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:01.107068    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:16:01.848434 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:01.848405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:01.848633 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:01.848449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:01.848633 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:01.848561    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:16:01.848633 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:01.848617    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret podName:693fe3c2-fe4c-4e69-9922-8b23c60e5d5e nodeName:}" failed. No retries permitted until 2026-09-11 18:16:17.84860289 +0000 UTC m=+33.243117928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret") pod "global-pull-secret-syncer-phxl4" (UID: "693fe3c2-fe4c-4e69-9922-8b23c60e5d5e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:16:01.848822 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:01.848561    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:16:01.848822 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:01.848683    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs podName:d24dd9d4-b46c-4d50-8d1e-7b494d53cb17 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:17.848671191 +0000 UTC m=+33.243186205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs") pod "network-metrics-daemon-tc5jj" (UID: "d24dd9d4-b46c-4d50-8d1e-7b494d53cb17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:16:01.949057 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:01.949021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fhw\" (UniqueName: \"kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw\") pod \"network-check-target-nw8l8\" (UID: \"0b26c0e1-d27d-4cb5-9731-8a321f327115\") " pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:01.949201 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:01.949164    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:16:01.949201 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:01.949178    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:16:01.949201 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:01.949187    2563 projected.go:196] Error preparing data for projected volume kube-api-access-h9fhw for pod openshift-network-diagnostics/network-check-target-nw8l8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:16:01.949355 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:01.949231    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw podName:0b26c0e1-d27d-4cb5-9731-8a321f327115 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:17.949220499 +0000 UTC m=+33.343735508 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h9fhw" (UniqueName: "kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw") pod "network-check-target-nw8l8" (UID: "0b26c0e1-d27d-4cb5-9731-8a321f327115") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:16:02.106518 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:02.106443    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:02.106659 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:02.106560    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:16:03.106893 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:03.106856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:03.107484 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:03.106855    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:03.107484 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:03.106972    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:16:03.107484 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:03.107061    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:16:04.107310 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:04.107276    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:04.107795 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:04.107404    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:16:05.108143 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.107992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:05.108486 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.108053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:05.108486 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:05.108245    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:16:05.108486 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:05.108272    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:16:05.370568 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.370281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-92.ec2.internal" event={"ID":"3d29890478aea68f0049d36cf8ba8661","Type":"ContainerStarted","Data":"5fa8daea4ed6bcf6d761bfb1d973b831e5be7660809c1ea707d995338169c85e"}
Sep 11 18:16:05.372061 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.372009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-948kd" event={"ID":"ee8197a9-86c5-472a-8d94-d7b0de84b73f","Type":"ContainerStarted","Data":"029256d7ac1d3017edc3a4e79aa3ada4b11885d83ad2306e6b2b3c7c73f98e3d"}
Sep 11 18:16:05.373795 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.373767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-489vf" event={"ID":"4f5a15bf-eecf-420f-a95f-bc0153e33007","Type":"ContainerStarted","Data":"78c81f141ba65a9ca20cee7b66c5588b808e683fe5761c8cec6ef6d624bd8729"}
Sep 11 18:16:05.376421 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.376402    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:16:05.376750 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.376726    2563 generic.go:356] "Generic (PLEG): container finished" podID="8248d578-0ff4-4992-b9b0-3327f5d8e663" containerID="28dbad55481ac35b694cff1310e9445278f2da97256a921d9d094ffbdfb7e8eb" exitCode=1
Sep 11 18:16:05.376812 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.376767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-442rc" event={"ID":"8248d578-0ff4-4992-b9b0-3327f5d8e663","Type":"ContainerStarted","Data":"f56eef9097222815c7cb20547f2c6a74945bd882512671e55e49aee3d6422a15"}
Sep 11 18:16:05.376852 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.376817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-442rc" event={"ID":"8248d578-0ff4-4992-b9b0-3327f5d8e663","Type":"ContainerStarted","Data":"1d055885e32709215b9567cd8013acf618a145849abfe62b3c905d997b8269c2"}
Sep 11 18:16:05.376884 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.376862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-442rc" event={"ID":"8248d578-0ff4-4992-b9b0-3327f5d8e663","Type":"ContainerStarted","Data":"00b4269d140d32a61db065c2dd89068fa4b20989dd1428e299d9603e457d583e"}
Sep 11 18:16:05.376918 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.376908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-442rc" event={"ID":"8248d578-0ff4-4992-b9b0-3327f5d8e663","Type":"ContainerDied","Data":"28dbad55481ac35b694cff1310e9445278f2da97256a921d9d094ffbdfb7e8eb"}
Sep 11 18:16:05.376985 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.376972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-442rc" event={"ID":"8248d578-0ff4-4992-b9b0-3327f5d8e663","Type":"ContainerStarted","Data":"f30bbe86b934c558ac990a638344a494fbca75ca6b24b40c0acbac467903fc49"}
Sep 11 18:16:05.389107 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.389061    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-92.ec2.internal" podStartSLOduration=19.389042242 podStartE2EDuration="19.389042242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:05.388559331 +0000 UTC m=+20.783074357" watchObservedRunningTime="2026-09-11 18:16:05.389042242 +0000 UTC m=+20.783557268"
Sep 11 18:16:05.433412 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.433342    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-948kd" podStartSLOduration=1.69511805 podStartE2EDuration="20.433328794s" totalImagesPullingTime="18.738210744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:05.433063654 +0000 UTC m=+20.827578714" watchObservedRunningTime="2026-09-11 18:16:05.433328794 +0000 UTC m=+20.827843817"
Sep 11 18:16:05.434123 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:05.434085    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-489vf" podStartSLOduration=2.110407272 podStartE2EDuration="20.434069067s" totalImagesPullingTime="18.323661795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:05.411500718 +0000 UTC m=+20.806015754" watchObservedRunningTime="2026-09-11 18:16:05.434069067 +0000 UTC m=+20.828584095"
Sep 11 18:16:06.107268 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.107239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:06.107432 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:06.107407    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:16:06.379530 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.379456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n8qwz" event={"ID":"cc5cc477-94d4-4770-8901-f1e8813f8ebc","Type":"ContainerStarted","Data":"bce9bf627aee4376d2e4464aa8e30035cd5afc20952c73f0777187b004375ec8"}
Sep 11 18:16:06.380976 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.380954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zkrtp" event={"ID":"9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0","Type":"ContainerStarted","Data":"82258d80b5764a6a2624c0d98e4ddea3b3586ced1c5cd463a20fe8789cee93b8"}
Sep 11 18:16:06.383806 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.383788    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:16:06.384169 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.384150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-442rc" event={"ID":"8248d578-0ff4-4992-b9b0-3327f5d8e663","Type":"ContainerStarted","Data":"3d6ad5a6b45af91ccf259bf83aeb859695ad199bef99948cdddc7409d94caf5f"}
Sep 11 18:16:06.385624 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.385605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4gfk7" event={"ID":"df16d6da-3b2d-4191-97af-de284cef6b8a","Type":"ContainerStarted","Data":"7e67f661623afe4466c3967589352ab82a077d1a068b434456656e070c81185a"}
Sep 11 18:16:06.387005 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.386981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qfkcq" event={"ID":"3725edf8-bf10-4f95-99cb-2dc2581932a4","Type":"ContainerStarted","Data":"29e6349a25866b4b194b0cbe847783c9ddc31c74fc60360894d10b8c0fc3eb3a"}
Sep 11 18:16:06.388331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.388312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj" event={"ID":"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0","Type":"ContainerStarted","Data":"dcaa482b988ce2af098b10c64db83392d51716a8876dfa10cc4b0648361a7ece"}
Sep 11 18:16:06.389717 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.389696    2563 generic.go:356] "Generic (PLEG): container finished" podID="e92ce411a910a52f68761a7f80b8212d" containerID="f4c96278a54d718e5a1d73601956627bb69966aed9fbdca27cc45252d18451ea" exitCode=0
Sep 11 18:16:06.389784 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.389769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal" event={"ID":"e92ce411a910a52f68761a7f80b8212d","Type":"ContainerDied","Data":"f4c96278a54d718e5a1d73601956627bb69966aed9fbdca27cc45252d18451ea"}
Sep 11 18:16:06.391299 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.391279    2563 generic.go:356] "Generic (PLEG): container finished" podID="13979e2d-de4f-4ee8-84ac-6595911215cc" containerID="1e8203f8a789e963ee7b9c94519b84239d8fd3e9f60de05a37e4b41ab59cb48c" exitCode=0
Sep 11 18:16:06.391401 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.391306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlqkh" event={"ID":"13979e2d-de4f-4ee8-84ac-6595911215cc","Type":"ContainerDied","Data":"1e8203f8a789e963ee7b9c94519b84239d8fd3e9f60de05a37e4b41ab59cb48c"}
Sep 11 18:16:06.394885 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.394848    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n8qwz" podStartSLOduration=3.200064159 podStartE2EDuration="21.394837124s" totalImagesPullingTime="18.194772965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:06.394668994 +0000 UTC m=+21.789184018" watchObservedRunningTime="2026-09-11 18:16:06.394837124 +0000 UTC m=+21.789352183"
Sep 11 18:16:06.411229 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.411196    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qfkcq" podStartSLOduration=3.110074814 podStartE2EDuration="21.411185281s" totalImagesPullingTime="18.301110467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:06.411021353 +0000 UTC m=+21.805536378" watchObservedRunningTime="2026-09-11 18:16:06.411185281 +0000 UTC m=+21.805700306"
Sep 11 18:16:06.469128 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.469088    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zkrtp" podStartSLOduration=3.177347728 podStartE2EDuration="21.469074739s" totalImagesPullingTime="18.291727011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:06.445605548 +0000 UTC m=+21.840120592" watchObservedRunningTime="2026-09-11 18:16:06.469074739 +0000 UTC m=+21.863589763"
Sep 11 18:16:06.485721 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.485692    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4gfk7" podStartSLOduration=6.86729914 podStartE2EDuration="21.485681735s" totalImagesPullingTime="14.618382595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:06.485428615 +0000 UTC m=+21.879943655" watchObservedRunningTime="2026-09-11 18:16:06.485681735 +0000 UTC m=+21.880196759"
Sep 11 18:16:06.591925 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.591794    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 18:16:06.629962 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.629873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:06.707810 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:06.707770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:07.079811 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:07.079705    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T18:16:06.591921151Z","UUID":"b79a090b-243c-4b3d-82fa-ea8c9d6eef7a","Handler":null,"Name":"","Endpoint":""}
Sep 11 18:16:07.081722 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:07.081699    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 18:16:07.081824 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:07.081733    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 18:16:07.107185 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:07.107166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:07.107272 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:07.107191    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:07.107272 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:07.107251    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:16:07.107462 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:07.107396    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:16:07.396564 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:07.396409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj" event={"ID":"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0","Type":"ContainerStarted","Data":"04fdc839220b632f170867955e553b0c213c6ab1d8e2ade042a27c7d4fc27139"}
Sep 11 18:16:07.398676 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:07.398647    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal" event={"ID":"e92ce411a910a52f68761a7f80b8212d","Type":"ContainerStarted","Data":"0e99ddc1cff81cf461eec6478511c5b5bf1298611f3d7a73275b26d085ebd688"}
Sep 11 18:16:07.834825 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:07.834794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:16:07.928068 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:07.928037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:07.998616 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:07.998553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:08.106919 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:08.106847    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:08.107043 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:08.106976    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:16:08.404666 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:08.404635    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:16:08.405136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:08.405113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-442rc" event={"ID":"8248d578-0ff4-4992-b9b0-3327f5d8e663","Type":"ContainerStarted","Data":"7a77a5dc6562ea061b7d7d7ea12a8072580dc58bfcec1e8077cdb9e6bd22a7f9"}
Sep 11 18:16:08.407492 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:08.407465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj" event={"ID":"b0d38a1d-e3f2-45d9-bdce-fb3530732ed0","Type":"ContainerStarted","Data":"cf6846ce2073d16e0a4e1fe099e90e18a3c986260199032e6d9062135c732b70"}
Sep 11 18:16:08.429959 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:08.429918    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtqjj" podStartSLOduration=3.204845885 podStartE2EDuration="23.429904499s" totalImagesPullingTime="20.225058614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:08.429872462 +0000 UTC m=+23.824387501" watchObservedRunningTime="2026-09-11 18:16:08.429904499 +0000 UTC m=+23.824419526"
Sep 11 18:16:08.430279 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:08.430249    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-92.ec2.internal" podStartSLOduration=22.430239851 podStartE2EDuration="22.430239851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:07.416739481 +0000 UTC m=+22.811254525" watchObservedRunningTime="2026-09-11 18:16:08.430239851 +0000 UTC m=+23.824754876"
Sep 11 18:16:09.106823 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:09.106794    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:09.107083 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:09.106847    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:09.107083 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:09.106927    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:16:09.107083 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:09.107057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:16:09.200667 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:09.200638    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:16:09.201204 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:09.201183    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:16:09.409962 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:09.409935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n8qwz"
Sep 11 18:16:09.476599 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:09.476572    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:09.839144 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:09.839068    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:10.106828 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:10.106757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:10.106992 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:10.106865    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:16:11.107546 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:11.107339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:11.107905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:11.107346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:11.107905 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:11.107594    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:16:11.107905 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:11.107667    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:16:11.414281 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:11.414255    2563 generic.go:356] "Generic (PLEG): container finished" podID="13979e2d-de4f-4ee8-84ac-6595911215cc" containerID="49c73ffc242e298e5b5ec860a9e4456cf6c39ab3c3b93c9b4cfda6ae91bf49b8" exitCode=0
Sep 11 18:16:11.414397 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:11.414357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlqkh" event={"ID":"13979e2d-de4f-4ee8-84ac-6595911215cc","Type":"ContainerDied","Data":"49c73ffc242e298e5b5ec860a9e4456cf6c39ab3c3b93c9b4cfda6ae91bf49b8"}
Sep 11 18:16:11.417982 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:11.417966    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:16:11.418334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:11.418312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-442rc" event={"ID":"8248d578-0ff4-4992-b9b0-3327f5d8e663","Type":"ContainerStarted","Data":"2bc9906ad1343cd5d860b7a5357e81b503d8b33c8942a58dc151e3b79a3a5947"}
Sep 11 18:16:11.418781 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:11.418763    2563 scope.go:117] "RemoveContainer" containerID="28dbad55481ac35b694cff1310e9445278f2da97256a921d9d094ffbdfb7e8eb"
Sep 11 18:16:11.678290 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:11.678198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:11.757175 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:11.757088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:12.107159 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.107134    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:12.107295 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:12.107264    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:16:12.356638 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.356467    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-phxl4"]
Sep 11 18:16:12.357079 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.356682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:12.357079 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:12.356754    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:16:12.378134 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.378111    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nw8l8"]
Sep 11 18:16:12.378212 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.378204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:12.378293 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:12.378274    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:16:12.380854 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.380782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc5jj"]
Sep 11 18:16:12.422748 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.422729    2563 generic.go:356] "Generic (PLEG): container finished" podID="13979e2d-de4f-4ee8-84ac-6595911215cc" containerID="ab243033fc57852cc44fa1f06775f5d6d44b2a23da9e19601d1307f66f521c79" exitCode=0
Sep 11 18:16:12.422846 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.422806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlqkh" event={"ID":"13979e2d-de4f-4ee8-84ac-6595911215cc","Type":"ContainerDied","Data":"ab243033fc57852cc44fa1f06775f5d6d44b2a23da9e19601d1307f66f521c79"}
Sep 11 18:16:12.426728 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.426713    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:16:12.427062 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.427039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-442rc" event={"ID":"8248d578-0ff4-4992-b9b0-3327f5d8e663","Type":"ContainerStarted","Data":"1be627dfe4bf703c48a3f1fae39b07079d847d91ac9cc5506fc3d474ccc47641"}
Sep 11 18:16:12.427287 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.427272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:12.427343 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.427294    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:16:12.427447 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.427355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:16:12.427447 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:12.427390    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:16:12.427447 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.427419    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:16:12.441622 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.441602    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:16:12.441740 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.441729    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:16:12.471331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:12.471296    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-442rc" podStartSLOduration=9.091220968 podStartE2EDuration="27.471284409s" totalImagesPullingTime="18.380063441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:12.471262715 +0000 UTC m=+27.865777742" watchObservedRunningTime="2026-09-11 18:16:12.471284409 +0000 UTC m=+27.865799470"
Sep 11 18:16:13.431612 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:13.431522    2563 generic.go:356] "Generic (PLEG): container finished" podID="13979e2d-de4f-4ee8-84ac-6595911215cc" containerID="bc92f16e0afbb4cba63692945fecd7be5a6beda9696c4a0b731bdd83de8aaf98" exitCode=0
Sep 11 18:16:13.432055 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:13.431603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlqkh" event={"ID":"13979e2d-de4f-4ee8-84ac-6595911215cc","Type":"ContainerDied","Data":"bc92f16e0afbb4cba63692945fecd7be5a6beda9696c4a0b731bdd83de8aaf98"}
Sep 11 18:16:14.106695 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:14.106616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:14.106695 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:14.106657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:14.106927 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:14.106625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:14.106927 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:14.106752    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:16:14.106927 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:14.106819    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:16:14.107123 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:14.106929    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:16:16.106537 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:16.106432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:16.107020 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:16.106572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:16.107020 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:16.106578    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phxl4" podUID="693fe3c2-fe4c-4e69-9922-8b23c60e5d5e"
Sep 11 18:16:16.107020 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:16.106685    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:16:16.107020 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:16.106732    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:16.107020 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:16.106784    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nw8l8" podUID="0b26c0e1-d27d-4cb5-9731-8a321f327115"
Sep 11 18:16:17.867116 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:17.867065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:17.867456 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:17.867115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:17.867456 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:17.867220    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:16:17.867456 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:17.867276    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret podName:693fe3c2-fe4c-4e69-9922-8b23c60e5d5e nodeName:}" failed. No retries permitted until 2026-09-11 18:16:49.867262424 +0000 UTC m=+65.261777427 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret") pod "global-pull-secret-syncer-phxl4" (UID: "693fe3c2-fe4c-4e69-9922-8b23c60e5d5e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:16:17.867456 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:17.867295    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:16:17.867456 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:17.867364    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs podName:d24dd9d4-b46c-4d50-8d1e-7b494d53cb17 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:49.867346035 +0000 UTC m=+65.261861040 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs") pod "network-metrics-daemon-tc5jj" (UID: "d24dd9d4-b46c-4d50-8d1e-7b494d53cb17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:16:17.924559 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:17.924542    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-92.ec2.internal" event="NodeReady"
Sep 11 18:16:17.924664 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:17.924658    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 18:16:17.967456 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:17.967432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fhw\" (UniqueName: \"kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw\") pod \"network-check-target-nw8l8\" (UID: \"0b26c0e1-d27d-4cb5-9731-8a321f327115\") " pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:17.967607 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:17.967588    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:16:17.967649 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:17.967612    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:16:17.967649 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:17.967622    2563 projected.go:196] Error preparing data for projected volume kube-api-access-h9fhw for pod openshift-network-diagnostics/network-check-target-nw8l8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:16:17.967722 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:17.967678    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw podName:0b26c0e1-d27d-4cb5-9731-8a321f327115 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:49.967666182 +0000 UTC m=+65.362181184 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h9fhw" (UniqueName: "kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw") pod "network-check-target-nw8l8" (UID: "0b26c0e1-d27d-4cb5-9731-8a321f327115") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:16:17.976401 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:17.976349    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d672m"]
Sep 11 18:16:17.988989 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:17.988969    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tw4p9"]
Sep 11 18:16:17.989100 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:17.989085    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:17.991699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:17.991670    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 18:16:17.991833 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:17.991816    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 18:16:17.991918 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:17.991903    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bf6d6\""
Sep 11 18:16:18.002984 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.002966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d672m"]
Sep 11 18:16:18.003074 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.002996    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tw4p9"]
Sep 11 18:16:18.003074 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.003057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tw4p9"
Sep 11 18:16:18.005322 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.005307    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gb2xv\""
Sep 11 18:16:18.005514 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.005498    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 18:16:18.005581 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.005554    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 18:16:18.005650 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.005633    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 18:16:18.068158 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.068137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8v6j\" (UniqueName: \"kubernetes.io/projected/f4fa4956-4afa-42b9-b95f-94cb9cc018df-kube-api-access-c8v6j\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.068236 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.068163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4fa4956-4afa-42b9-b95f-94cb9cc018df-tmp-dir\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.068236 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.068195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.068321 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.068301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fa4956-4afa-42b9-b95f-94cb9cc018df-config-volume\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.106675 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.106650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:18.106795 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.106654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:18.106854 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.106654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:18.110114 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.110099    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:16:18.110358 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.110341    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:16:18.110439 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.110353    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:16:18.110439 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.110413    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-76gd9\""
Sep 11 18:16:18.110584 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.110566    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:16:18.110649 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.110638    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bn27r\""
Sep 11 18:16:18.137691 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.137674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:18.168520 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.168500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4fa4956-4afa-42b9-b95f-94cb9cc018df-tmp-dir\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.168593 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.168538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48qvs\" (UniqueName: \"kubernetes.io/projected/4e8d1a68-a884-44ec-b9cc-f6ad7f74156c-kube-api-access-48qvs\") pod \"ingress-canary-tw4p9\" (UID: \"4e8d1a68-a884-44ec-b9cc-f6ad7f74156c\") " pod="openshift-ingress-canary/ingress-canary-tw4p9"
Sep 11 18:16:18.168593 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.168558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.168705 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.168645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fa4956-4afa-42b9-b95f-94cb9cc018df-config-volume\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.168705 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.168675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e8d1a68-a884-44ec-b9cc-f6ad7f74156c-cert\") pod \"ingress-canary-tw4p9\" (UID: \"4e8d1a68-a884-44ec-b9cc-f6ad7f74156c\") " pod="openshift-ingress-canary/ingress-canary-tw4p9"
Sep 11 18:16:18.168775 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:18.168710    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:18.168775 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:18.168751    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls podName:f4fa4956-4afa-42b9-b95f-94cb9cc018df nodeName:}" failed. No retries permitted until 2026-09-11 18:16:18.668739133 +0000 UTC m=+34.063254136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls") pod "dns-default-d672m" (UID: "f4fa4956-4afa-42b9-b95f-94cb9cc018df") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:18.168775 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.168709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8v6j\" (UniqueName: \"kubernetes.io/projected/f4fa4956-4afa-42b9-b95f-94cb9cc018df-kube-api-access-c8v6j\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.168902 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.168827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4fa4956-4afa-42b9-b95f-94cb9cc018df-tmp-dir\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.169151 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.169136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fa4956-4afa-42b9-b95f-94cb9cc018df-config-volume\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.181425 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.181397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8v6j\" (UniqueName: \"kubernetes.io/projected/f4fa4956-4afa-42b9-b95f-94cb9cc018df-kube-api-access-c8v6j\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.213361 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.213228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:18.269157 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.269136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48qvs\" (UniqueName: \"kubernetes.io/projected/4e8d1a68-a884-44ec-b9cc-f6ad7f74156c-kube-api-access-48qvs\") pod \"ingress-canary-tw4p9\" (UID: \"4e8d1a68-a884-44ec-b9cc-f6ad7f74156c\") " pod="openshift-ingress-canary/ingress-canary-tw4p9"
Sep 11 18:16:18.269245 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.269196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e8d1a68-a884-44ec-b9cc-f6ad7f74156c-cert\") pod \"ingress-canary-tw4p9\" (UID: \"4e8d1a68-a884-44ec-b9cc-f6ad7f74156c\") " pod="openshift-ingress-canary/ingress-canary-tw4p9"
Sep 11 18:16:18.271450 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.271433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e8d1a68-a884-44ec-b9cc-f6ad7f74156c-cert\") pod \"ingress-canary-tw4p9\" (UID: \"4e8d1a68-a884-44ec-b9cc-f6ad7f74156c\") " pod="openshift-ingress-canary/ingress-canary-tw4p9"
Sep 11 18:16:18.278795 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.278774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48qvs\" (UniqueName: \"kubernetes.io/projected/4e8d1a68-a884-44ec-b9cc-f6ad7f74156c-kube-api-access-48qvs\") pod \"ingress-canary-tw4p9\" (UID: \"4e8d1a68-a884-44ec-b9cc-f6ad7f74156c\") " pod="openshift-ingress-canary/ingress-canary-tw4p9"
Sep 11 18:16:18.311862 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.311843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tw4p9"
Sep 11 18:16:18.313430 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.313391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:18.445893 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.445865    2563 generic.go:356] "Generic (PLEG): container finished" podID="13979e2d-de4f-4ee8-84ac-6595911215cc" containerID="dd601701fb47dcbf19af726ed627f1939df844af9e8d6eb569bec098f1c44edf" exitCode=0
Sep 11 18:16:18.446040 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.445939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlqkh" event={"ID":"13979e2d-de4f-4ee8-84ac-6595911215cc","Type":"ContainerDied","Data":"dd601701fb47dcbf19af726ed627f1939df844af9e8d6eb569bec098f1c44edf"}
Sep 11 18:16:18.616262 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.616239    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tw4p9"]
Sep 11 18:16:18.654101 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:16:18.654075    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8d1a68_a884_44ec_b9cc_f6ad7f74156c.slice/crio-6341fd73225d1968097f1f90d76e688009ca4b6f76426baed4a95e54e789680f WatchSource:0}: Error finding container 6341fd73225d1968097f1f90d76e688009ca4b6f76426baed4a95e54e789680f: Status 404 returned error can't find the container with id 6341fd73225d1968097f1f90d76e688009ca4b6f76426baed4a95e54e789680f
Sep 11 18:16:18.674767 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:18.672012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:18.674767 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:18.672415    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:18.674767 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:18.672496    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls podName:f4fa4956-4afa-42b9-b95f-94cb9cc018df nodeName:}" failed. No retries permitted until 2026-09-11 18:16:19.672465424 +0000 UTC m=+35.066980442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls") pod "dns-default-d672m" (UID: "f4fa4956-4afa-42b9-b95f-94cb9cc018df") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:19.450491 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:19.450442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tw4p9" event={"ID":"4e8d1a68-a884-44ec-b9cc-f6ad7f74156c","Type":"ContainerStarted","Data":"6341fd73225d1968097f1f90d76e688009ca4b6f76426baed4a95e54e789680f"}
Sep 11 18:16:19.454091 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:19.454060    2563 generic.go:356] "Generic (PLEG): container finished" podID="13979e2d-de4f-4ee8-84ac-6595911215cc" containerID="72ef012d6a14ddb493e4de8dd40ed691ff3957ce974b6c95818905da29c88180" exitCode=0
Sep 11 18:16:19.454221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:19.454108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlqkh" event={"ID":"13979e2d-de4f-4ee8-84ac-6595911215cc","Type":"ContainerDied","Data":"72ef012d6a14ddb493e4de8dd40ed691ff3957ce974b6c95818905da29c88180"}
Sep 11 18:16:19.679399 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:19.679350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:19.679507 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:19.679488    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:19.679558 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:19.679550    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls podName:f4fa4956-4afa-42b9-b95f-94cb9cc018df nodeName:}" failed. No retries permitted until 2026-09-11 18:16:21.679530403 +0000 UTC m=+37.074045423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls") pod "dns-default-d672m" (UID: "f4fa4956-4afa-42b9-b95f-94cb9cc018df") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:20.461737 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:20.461702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlqkh" event={"ID":"13979e2d-de4f-4ee8-84ac-6595911215cc","Type":"ContainerStarted","Data":"3fdcea3fd9ebffe3ee7b5bf5c928fc653b7ae37b9b1933ee2655a892a80fb131"}
Sep 11 18:16:20.488891 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:20.488841    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qlqkh" podStartSLOduration=7.271428769 podStartE2EDuration="35.488827108s" totalImagesPullingTime="28.217398339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:20.487247134 +0000 UTC m=+35.881762160" watchObservedRunningTime="2026-09-11 18:16:20.488827108 +0000 UTC m=+35.883342132"
Sep 11 18:16:21.465876 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:21.465702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tw4p9" event={"ID":"4e8d1a68-a884-44ec-b9cc-f6ad7f74156c","Type":"ContainerStarted","Data":"8854f2c3d37473f236852576659c68ca7085a1a97b3d0845ed385a357eba367e"}
Sep 11 18:16:21.481835 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:21.481793    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tw4p9" podStartSLOduration=2.259159322 podStartE2EDuration="4.48177905s" totalImagesPullingTime="2.222619728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:21.481429217 +0000 UTC m=+36.875944255" watchObservedRunningTime="2026-09-11 18:16:21.48177905 +0000 UTC m=+36.876294070"
Sep 11 18:16:21.693854 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:21.693833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:21.693969 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:21.693956    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:21.694017 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:21.694008    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls podName:f4fa4956-4afa-42b9-b95f-94cb9cc018df nodeName:}" failed. No retries permitted until 2026-09-11 18:16:25.693991424 +0000 UTC m=+41.088506438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls") pod "dns-default-d672m" (UID: "f4fa4956-4afa-42b9-b95f-94cb9cc018df") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:25.722956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:25.722925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:25.723276 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:25.723061    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:25.723276 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:25.723131    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls podName:f4fa4956-4afa-42b9-b95f-94cb9cc018df nodeName:}" failed. No retries permitted until 2026-09-11 18:16:33.723114784 +0000 UTC m=+49.117629794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls") pod "dns-default-d672m" (UID: "f4fa4956-4afa-42b9-b95f-94cb9cc018df") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:33.778084 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:33.778050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:33.778476 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:33.778196    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:33.778476 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:33.778252    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls podName:f4fa4956-4afa-42b9-b95f-94cb9cc018df nodeName:}" failed. No retries permitted until 2026-09-11 18:16:49.778238828 +0000 UTC m=+65.172753830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls") pod "dns-default-d672m" (UID: "f4fa4956-4afa-42b9-b95f-94cb9cc018df") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:44.444900 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:44.444869    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-442rc"
Sep 11 18:16:49.779262 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:49.779227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:16:49.779628 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:49.779318    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:49.779628 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:49.779386    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls podName:f4fa4956-4afa-42b9-b95f-94cb9cc018df nodeName:}" failed. No retries permitted until 2026-09-11 18:17:21.779360896 +0000 UTC m=+97.173875900 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls") pod "dns-default-d672m" (UID: "f4fa4956-4afa-42b9-b95f-94cb9cc018df") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:49.879684 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:49.879655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:16:49.879763 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:49.879696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:49.882382 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:49.882332    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:16:49.882578 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:49.882563    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:16:49.890630 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:49.890613    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 18:16:49.890690 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:16:49.890674    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs podName:d24dd9d4-b46c-4d50-8d1e-7b494d53cb17 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:53.890660997 +0000 UTC m=+129.285176005 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs") pod "network-metrics-daemon-tc5jj" (UID: "d24dd9d4-b46c-4d50-8d1e-7b494d53cb17") : secret "metrics-daemon-secret" not found
Sep 11 18:16:49.893497 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:49.893479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/693fe3c2-fe4c-4e69-9922-8b23c60e5d5e-original-pull-secret\") pod \"global-pull-secret-syncer-phxl4\" (UID: \"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e\") " pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:49.915258 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:49.915238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phxl4"
Sep 11 18:16:49.980821 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:49.980796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fhw\" (UniqueName: \"kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw\") pod \"network-check-target-nw8l8\" (UID: \"0b26c0e1-d27d-4cb5-9731-8a321f327115\") " pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:49.983698 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:49.983674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:16:49.993451 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:49.993436    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:16:50.003749 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:50.003731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fhw\" (UniqueName: \"kubernetes.io/projected/0b26c0e1-d27d-4cb5-9731-8a321f327115-kube-api-access-h9fhw\") pod \"network-check-target-nw8l8\" (UID: \"0b26c0e1-d27d-4cb5-9731-8a321f327115\") " pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:50.181744 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:50.181725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-phxl4"]
Sep 11 18:16:50.183992 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:16:50.183972    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693fe3c2_fe4c_4e69_9922_8b23c60e5d5e.slice/crio-6a06ff55d0c8d59cafb8d9404295763b58315db3296f2f556416f215cdaadb73 WatchSource:0}: Error finding container 6a06ff55d0c8d59cafb8d9404295763b58315db3296f2f556416f215cdaadb73: Status 404 returned error can't find the container with id 6a06ff55d0c8d59cafb8d9404295763b58315db3296f2f556416f215cdaadb73
Sep 11 18:16:50.229434 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:50.229415    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-76gd9\""
Sep 11 18:16:50.236936 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:50.236921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:50.500504 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:50.500414    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nw8l8"]
Sep 11 18:16:50.503426 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:16:50.503402    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b26c0e1_d27d_4cb5_9731_8a321f327115.slice/crio-a1a6d98e119cfdcb9eed3a9bdaeab1382c45b11f438e765be256ce30c93194d2 WatchSource:0}: Error finding container a1a6d98e119cfdcb9eed3a9bdaeab1382c45b11f438e765be256ce30c93194d2: Status 404 returned error can't find the container with id a1a6d98e119cfdcb9eed3a9bdaeab1382c45b11f438e765be256ce30c93194d2
Sep 11 18:16:50.527042 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:50.527013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nw8l8" event={"ID":"0b26c0e1-d27d-4cb5-9731-8a321f327115","Type":"ContainerStarted","Data":"a1a6d98e119cfdcb9eed3a9bdaeab1382c45b11f438e765be256ce30c93194d2"}
Sep 11 18:16:50.528084 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:50.528064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-phxl4" event={"ID":"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e","Type":"ContainerStarted","Data":"6a06ff55d0c8d59cafb8d9404295763b58315db3296f2f556416f215cdaadb73"}
Sep 11 18:16:55.542203 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:55.542168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nw8l8" event={"ID":"0b26c0e1-d27d-4cb5-9731-8a321f327115","Type":"ContainerStarted","Data":"8feb38caaf9fa73c042d522af0647ed503a0252316de5a8a879eab1d6ccfeb60"}
Sep 11 18:16:55.542629 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:55.542292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:16:55.543511 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:55.543494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-phxl4" event={"ID":"693fe3c2-fe4c-4e69-9922-8b23c60e5d5e","Type":"ContainerStarted","Data":"1dbdbaaedb7a5752af63bd91a34ad77246c884f68c7012e9efe4adbfe8216b5f"}
Sep 11 18:16:55.562069 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:55.562025    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nw8l8" podStartSLOduration=66.338784753 podStartE2EDuration="1m10.562012964s" totalImagesPullingTime="4.223228211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:55.561968553 +0000 UTC m=+70.956483588" watchObservedRunningTime="2026-09-11 18:16:55.562012964 +0000 UTC m=+70.956527988"
Sep 11 18:16:55.581141 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:16:55.581103    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-phxl4" podStartSLOduration=66.033809611 podStartE2EDuration="1m10.581091686s" totalImagesPullingTime="4.547282075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:55.579024144 +0000 UTC m=+70.973539169" watchObservedRunningTime="2026-09-11 18:16:55.581091686 +0000 UTC m=+70.975606711"
Sep 11 18:17:21.779899 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:17:21.779788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:17:21.780347 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:17:21.779903    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:17:21.780347 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:17:21.779960    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls podName:f4fa4956-4afa-42b9-b95f-94cb9cc018df nodeName:}" failed. No retries permitted until 2026-09-11 18:18:25.779947404 +0000 UTC m=+161.174462406 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls") pod "dns-default-d672m" (UID: "f4fa4956-4afa-42b9-b95f-94cb9cc018df") : secret "dns-default-metrics-tls" not found
Sep 11 18:17:26.548340 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:17:26.548308    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nw8l8"
Sep 11 18:17:53.982976 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:17:53.982936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:17:53.983524 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:17:53.983064    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 18:17:53.983524 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:17:53.983145    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs podName:d24dd9d4-b46c-4d50-8d1e-7b494d53cb17 nodeName:}" failed. No retries permitted until 2026-09-11 18:19:55.98312481 +0000 UTC m=+251.377639831 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs") pod "network-metrics-daemon-tc5jj" (UID: "d24dd9d4-b46c-4d50-8d1e-7b494d53cb17") : secret "metrics-daemon-secret" not found
Sep 11 18:18:00.248293 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.248258    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-bmtcc"]
Sep 11 18:18:00.251075 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.251058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.253686 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.253662    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 18:18:00.253795 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.253780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 18:18:00.253860 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.253804    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 18:18:00.255177 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.255157    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 18:18:00.255284 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.255185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-99klh\""
Sep 11 18:18:00.260831 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.260814    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 18:18:00.268398 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.268364    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-bmtcc"]
Sep 11 18:18:00.324175 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.324150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f7d9182c-e4a7-4850-b8f9-a43199335b21-snapshots\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.324258 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.324179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d9182c-e4a7-4850-b8f9-a43199335b21-trusted-ca-bundle\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.324258 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.324197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d9182c-e4a7-4850-b8f9-a43199335b21-service-ca-bundle\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.324258 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.324227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7d9182c-e4a7-4850-b8f9-a43199335b21-tmp\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.324360 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.324281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlgf\" (UniqueName: \"kubernetes.io/projected/f7d9182c-e4a7-4850-b8f9-a43199335b21-kube-api-access-mhlgf\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.324360 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.324314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7d9182c-e4a7-4850-b8f9-a43199335b21-serving-cert\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.358273 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.358250    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5dbccf7754-qnh7w"]
Sep 11 18:18:00.360932 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.360916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.363417 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.363397    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 18:18:00.363517 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.363500    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lqbxx\""
Sep 11 18:18:00.363579 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.363537    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 18:18:00.363646 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.363629    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 18:18:00.363761 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.363715    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 18:18:00.363902 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.363797    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 18:18:00.363975 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.363913    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 18:18:00.375577 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.375547    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5dbccf7754-qnh7w"]
Sep 11 18:18:00.425533 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.425506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-default-certificate\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.425625 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.425539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlgf\" (UniqueName: \"kubernetes.io/projected/f7d9182c-e4a7-4850-b8f9-a43199335b21-kube-api-access-mhlgf\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.425625 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.425565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-stats-auth\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.425625 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.425617    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.425782 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.425648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-224hv\" (UniqueName: \"kubernetes.io/projected/4c33a960-4be6-4055-9855-b13ccf170c79-kube-api-access-224hv\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.425782 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.425685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7d9182c-e4a7-4850-b8f9-a43199335b21-serving-cert\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.425782 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.425703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f7d9182c-e4a7-4850-b8f9-a43199335b21-snapshots\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.425782 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.425717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.425782 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.425744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d9182c-e4a7-4850-b8f9-a43199335b21-trusted-ca-bundle\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.425782 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.425779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d9182c-e4a7-4850-b8f9-a43199335b21-service-ca-bundle\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.426019 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.425827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7d9182c-e4a7-4850-b8f9-a43199335b21-tmp\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.426242 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.426214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7d9182c-e4a7-4850-b8f9-a43199335b21-tmp\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.426313 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.426297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f7d9182c-e4a7-4850-b8f9-a43199335b21-snapshots\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.426361 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.426316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d9182c-e4a7-4850-b8f9-a43199335b21-service-ca-bundle\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.426683 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.426664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d9182c-e4a7-4850-b8f9-a43199335b21-trusted-ca-bundle\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.428138 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.428111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7d9182c-e4a7-4850-b8f9-a43199335b21-serving-cert\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.435045 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.435024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlgf\" (UniqueName: \"kubernetes.io/projected/f7d9182c-e4a7-4850-b8f9-a43199335b21-kube-api-access-mhlgf\") pod \"insights-operator-5844b856db-bmtcc\" (UID: \"f7d9182c-e4a7-4850-b8f9-a43199335b21\") " pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.460685 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.460665    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"]
Sep 11 18:18:00.463750 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.463737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:00.470505 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.470487    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kv5k9\""
Sep 11 18:18:00.470610 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.470494    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 18:18:00.470700 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.470686    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:18:00.471114 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.471099    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 18:18:00.481110 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.481091    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"]
Sep 11 18:18:00.526673 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.526626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vpl2x\" (UID: \"87da72fc-2384-4c58-a47b-06ec0d24de09\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:00.526751 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.526677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-default-certificate\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.526751 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.526726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-stats-auth\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.526825 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.526754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.526825 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.526775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-224hv\" (UniqueName: \"kubernetes.io/projected/4c33a960-4be6-4055-9855-b13ccf170c79-kube-api-access-224hv\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.526825 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.526811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2nn7\" (UniqueName: \"kubernetes.io/projected/87da72fc-2384-4c58-a47b-06ec0d24de09-kube-api-access-t2nn7\") pod \"cluster-samples-operator-86499c89c4-vpl2x\" (UID: \"87da72fc-2384-4c58-a47b-06ec0d24de09\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:00.526935 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.526840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.526935 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:00.526920    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:01.02690175 +0000 UTC m=+136.421416772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : configmap references non-existent config key: service-ca.crt
Sep 11 18:18:00.526935 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:00.526928    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 18:18:00.527102 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:00.526974    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:01.026962429 +0000 UTC m=+136.421477450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : secret "router-metrics-certs-default" not found
Sep 11 18:18:00.528825 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.528808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-stats-auth\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.528985 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.528965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-default-certificate\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.536549 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.536524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-224hv\" (UniqueName: \"kubernetes.io/projected/4c33a960-4be6-4055-9855-b13ccf170c79-kube-api-access-224hv\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:00.559457 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.559438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-bmtcc"
Sep 11 18:18:00.560725 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.560704    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-lgjg8"]
Sep 11 18:18:00.563254 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.563236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.566319 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.566296    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 11 18:18:00.566421 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.566406    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:18:00.566657 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.566636    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 11 18:18:00.566888 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.566871    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-r7g8g\""
Sep 11 18:18:00.566960 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.566873    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 11 18:18:00.573224 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.573204    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 11 18:18:00.577228 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.577198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-lgjg8"]
Sep 11 18:18:00.627758 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.627724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7lgt\" (UniqueName: \"kubernetes.io/projected/fdd54bfd-280d-48a9-bdaf-4a38738c6856-kube-api-access-w7lgt\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.627859 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.627768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2nn7\" (UniqueName: \"kubernetes.io/projected/87da72fc-2384-4c58-a47b-06ec0d24de09-kube-api-access-t2nn7\") pod \"cluster-samples-operator-86499c89c4-vpl2x\" (UID: \"87da72fc-2384-4c58-a47b-06ec0d24de09\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:00.627859 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.627786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdd54bfd-280d-48a9-bdaf-4a38738c6856-trusted-ca\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.627859 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.627853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vpl2x\" (UID: \"87da72fc-2384-4c58-a47b-06ec0d24de09\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:00.628002 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.627913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdd54bfd-280d-48a9-bdaf-4a38738c6856-config\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.628002 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.627932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdd54bfd-280d-48a9-bdaf-4a38738c6856-serving-cert\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.628091 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:00.627994    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:18:00.628091 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:00.628048    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls podName:87da72fc-2384-4c58-a47b-06ec0d24de09 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:01.128034075 +0000 UTC m=+136.522549108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-vpl2x" (UID: "87da72fc-2384-4c58-a47b-06ec0d24de09") : secret "samples-operator-tls" not found
Sep 11 18:18:00.638628 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.638607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2nn7\" (UniqueName: \"kubernetes.io/projected/87da72fc-2384-4c58-a47b-06ec0d24de09-kube-api-access-t2nn7\") pod \"cluster-samples-operator-86499c89c4-vpl2x\" (UID: \"87da72fc-2384-4c58-a47b-06ec0d24de09\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:00.728445 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.728416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdd54bfd-280d-48a9-bdaf-4a38738c6856-config\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.728445 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.728444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdd54bfd-280d-48a9-bdaf-4a38738c6856-serving-cert\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.728577 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.728475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7lgt\" (UniqueName: \"kubernetes.io/projected/fdd54bfd-280d-48a9-bdaf-4a38738c6856-kube-api-access-w7lgt\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.728577 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.728491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdd54bfd-280d-48a9-bdaf-4a38738c6856-trusted-ca\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.729080 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.729061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdd54bfd-280d-48a9-bdaf-4a38738c6856-config\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.729159 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.729144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdd54bfd-280d-48a9-bdaf-4a38738c6856-trusted-ca\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.730502 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.730483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdd54bfd-280d-48a9-bdaf-4a38738c6856-serving-cert\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.736135 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.736119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7lgt\" (UniqueName: \"kubernetes.io/projected/fdd54bfd-280d-48a9-bdaf-4a38738c6856-kube-api-access-w7lgt\") pod \"console-operator-696884fb9c-lgjg8\" (UID: \"fdd54bfd-280d-48a9-bdaf-4a38738c6856\") " pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:00.828475 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.828423    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-bmtcc"]
Sep 11 18:18:00.831031 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:00.831001    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d9182c_e4a7_4850_b8f9_a43199335b21.slice/crio-a8f1dc6b383601c898b338caf0587051f65e4cdafa948956edf9f7e74dd10dd9 WatchSource:0}: Error finding container a8f1dc6b383601c898b338caf0587051f65e4cdafa948956edf9f7e74dd10dd9: Status 404 returned error can't find the container with id a8f1dc6b383601c898b338caf0587051f65e4cdafa948956edf9f7e74dd10dd9
Sep 11 18:18:00.873022 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:00.872998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:01.031617 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:01.031591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:01.031718 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:01.031671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:01.031768 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:01.031750    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 18:18:01.031810 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:01.031800    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:02.031785161 +0000 UTC m=+137.426300179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : secret "router-metrics-certs-default" not found
Sep 11 18:18:01.031853 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:01.031818    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:02.031810889 +0000 UTC m=+137.426325891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : configmap references non-existent config key: service-ca.crt
Sep 11 18:18:01.132160 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:01.132108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vpl2x\" (UID: \"87da72fc-2384-4c58-a47b-06ec0d24de09\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:01.132257 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:01.132206    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:18:01.132257 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:01.132254    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls podName:87da72fc-2384-4c58-a47b-06ec0d24de09 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:02.132243171 +0000 UTC m=+137.526758173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-vpl2x" (UID: "87da72fc-2384-4c58-a47b-06ec0d24de09") : secret "samples-operator-tls" not found
Sep 11 18:18:01.140641 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:01.140616    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-lgjg8"]
Sep 11 18:18:01.144026 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:01.143997    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd54bfd_280d_48a9_bdaf_4a38738c6856.slice/crio-ee902c8e1f750ff653bcc4ec9b07f1f495ccb6d64b8c2b99e5a0a5722ab6449f WatchSource:0}: Error finding container ee902c8e1f750ff653bcc4ec9b07f1f495ccb6d64b8c2b99e5a0a5722ab6449f: Status 404 returned error can't find the container with id ee902c8e1f750ff653bcc4ec9b07f1f495ccb6d64b8c2b99e5a0a5722ab6449f
Sep 11 18:18:01.685361 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:01.685328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" event={"ID":"fdd54bfd-280d-48a9-bdaf-4a38738c6856","Type":"ContainerStarted","Data":"ee902c8e1f750ff653bcc4ec9b07f1f495ccb6d64b8c2b99e5a0a5722ab6449f"}
Sep 11 18:18:01.686805 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:01.686781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-bmtcc" event={"ID":"f7d9182c-e4a7-4850-b8f9-a43199335b21","Type":"ContainerStarted","Data":"a8f1dc6b383601c898b338caf0587051f65e4cdafa948956edf9f7e74dd10dd9"}
Sep 11 18:18:02.037033 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:02.036949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:02.037033 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:02.036995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:02.037209 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:02.037094    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 18:18:02.037209 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:02.037134    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:04.037111915 +0000 UTC m=+139.431626934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : configmap references non-existent config key: service-ca.crt
Sep 11 18:18:02.037209 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:02.037163    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:04.037152627 +0000 UTC m=+139.431667643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : secret "router-metrics-certs-default" not found
Sep 11 18:18:02.137873 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:02.137844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vpl2x\" (UID: \"87da72fc-2384-4c58-a47b-06ec0d24de09\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:02.138040 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:02.137964    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:18:02.138040 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:02.138026    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls podName:87da72fc-2384-4c58-a47b-06ec0d24de09 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:04.138009782 +0000 UTC m=+139.532524798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-vpl2x" (UID: "87da72fc-2384-4c58-a47b-06ec0d24de09") : secret "samples-operator-tls" not found
Sep 11 18:18:02.590740 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:02.590703    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-rhlvr"]
Sep 11 18:18:02.595293 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:02.595264    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-rhlvr"
Sep 11 18:18:02.602352 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:02.598512    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gf4km\""
Sep 11 18:18:02.603763 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:02.603720    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-rhlvr"]
Sep 11 18:18:02.642287 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:02.642261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zffb6\" (UniqueName: \"kubernetes.io/projected/0386cb0f-54d0-4398-8d7d-0f04475093a0-kube-api-access-zffb6\") pod \"network-check-source-6f8b768455-rhlvr\" (UID: \"0386cb0f-54d0-4398-8d7d-0f04475093a0\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-rhlvr"
Sep 11 18:18:02.743555 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:02.743527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zffb6\" (UniqueName: \"kubernetes.io/projected/0386cb0f-54d0-4398-8d7d-0f04475093a0-kube-api-access-zffb6\") pod \"network-check-source-6f8b768455-rhlvr\" (UID: \"0386cb0f-54d0-4398-8d7d-0f04475093a0\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-rhlvr"
Sep 11 18:18:02.753747 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:02.753726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zffb6\" (UniqueName: \"kubernetes.io/projected/0386cb0f-54d0-4398-8d7d-0f04475093a0-kube-api-access-zffb6\") pod \"network-check-source-6f8b768455-rhlvr\" (UID: \"0386cb0f-54d0-4398-8d7d-0f04475093a0\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-rhlvr"
Sep 11 18:18:02.910165 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:02.910134    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-rhlvr"
Sep 11 18:18:03.694743 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:03.694718    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/0.log"
Sep 11 18:18:03.694848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:03.694756    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdd54bfd-280d-48a9-bdaf-4a38738c6856" containerID="4f19ae067b12036b8196d7984a1e46963287abf08319254b2e758932d19b237c" exitCode=255
Sep 11 18:18:03.694848 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:03.694822    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" event={"ID":"fdd54bfd-280d-48a9-bdaf-4a38738c6856","Type":"ContainerDied","Data":"4f19ae067b12036b8196d7984a1e46963287abf08319254b2e758932d19b237c"}
Sep 11 18:18:03.695070 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:03.695054    2563 scope.go:117] "RemoveContainer" containerID="4f19ae067b12036b8196d7984a1e46963287abf08319254b2e758932d19b237c"
Sep 11 18:18:03.696423 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:03.696399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-bmtcc" event={"ID":"f7d9182c-e4a7-4850-b8f9-a43199335b21","Type":"ContainerStarted","Data":"ca32338f46db50ebe69689bced8171ed5b6192883eae8179a119c975d98059ab"}
Sep 11 18:18:03.757423 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:03.757280    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-bmtcc" podStartSLOduration=1.085826073 podStartE2EDuration="3.75726218s" totalImagesPullingTime="2.671436107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:03.756699528 +0000 UTC m=+139.151214553" watchObservedRunningTime="2026-09-11 18:18:03.75726218 +0000 UTC m=+139.151777206"
Sep 11 18:18:03.770357 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:03.770284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-rhlvr"]
Sep 11 18:18:03.774419 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:03.774353    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0386cb0f_54d0_4398_8d7d_0f04475093a0.slice/crio-c6db85f48226a31474ead33f287c3509bf975cf2547a35e4204b74b3990595ae WatchSource:0}: Error finding container c6db85f48226a31474ead33f287c3509bf975cf2547a35e4204b74b3990595ae: Status 404 returned error can't find the container with id c6db85f48226a31474ead33f287c3509bf975cf2547a35e4204b74b3990595ae
Sep 11 18:18:04.054687 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.054615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:04.054687 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.054673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:04.054877 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:04.054769    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:08.054751919 +0000 UTC m=+143.449266922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : configmap references non-existent config key: service-ca.crt
Sep 11 18:18:04.054877 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:04.054797    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 18:18:04.054877 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:04.054843    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:08.054829108 +0000 UTC m=+143.449344115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : secret "router-metrics-certs-default" not found
Sep 11 18:18:04.155232 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.155206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vpl2x\" (UID: \"87da72fc-2384-4c58-a47b-06ec0d24de09\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:04.155365 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:04.155349    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:18:04.155440 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:04.155416    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls podName:87da72fc-2384-4c58-a47b-06ec0d24de09 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:08.155399716 +0000 UTC m=+143.549914737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-vpl2x" (UID: "87da72fc-2384-4c58-a47b-06ec0d24de09") : secret "samples-operator-tls" not found
Sep 11 18:18:04.701593 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.701572    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/1.log"
Sep 11 18:18:04.702039 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.702020    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/0.log"
Sep 11 18:18:04.702103 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.702064    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdd54bfd-280d-48a9-bdaf-4a38738c6856" containerID="defc8fe71245cbc01f628864176bb63cd26487a204b7d3b2f9837ac014a226a3" exitCode=255
Sep 11 18:18:04.702153 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.702141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" event={"ID":"fdd54bfd-280d-48a9-bdaf-4a38738c6856","Type":"ContainerDied","Data":"defc8fe71245cbc01f628864176bb63cd26487a204b7d3b2f9837ac014a226a3"}
Sep 11 18:18:04.702214 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.702201    2563 scope.go:117] "RemoveContainer" containerID="4f19ae067b12036b8196d7984a1e46963287abf08319254b2e758932d19b237c"
Sep 11 18:18:04.702449 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.702418    2563 scope.go:117] "RemoveContainer" containerID="defc8fe71245cbc01f628864176bb63cd26487a204b7d3b2f9837ac014a226a3"
Sep 11 18:18:04.702648 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:04.702591    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-lgjg8_openshift-console-operator(fdd54bfd-280d-48a9-bdaf-4a38738c6856)\"" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" podUID="fdd54bfd-280d-48a9-bdaf-4a38738c6856"
Sep 11 18:18:04.703800 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.703779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-rhlvr" event={"ID":"0386cb0f-54d0-4398-8d7d-0f04475093a0","Type":"ContainerStarted","Data":"f3867df58929a19f65784ed4ffae8dd9e9c0b5825a8364e9799b0637be2e8edc"}
Sep 11 18:18:04.703885 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.703821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-rhlvr" event={"ID":"0386cb0f-54d0-4398-8d7d-0f04475093a0","Type":"ContainerStarted","Data":"c6db85f48226a31474ead33f287c3509bf975cf2547a35e4204b74b3990595ae"}
Sep 11 18:18:04.744853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:04.744810    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-rhlvr" podStartSLOduration=2.744795237 podStartE2EDuration="2.744795237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:04.744351141 +0000 UTC m=+140.138866183" watchObservedRunningTime="2026-09-11 18:18:04.744795237 +0000 UTC m=+140.139310264"
Sep 11 18:18:05.029003 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.028937    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h"]
Sep 11 18:18:05.033665 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.033642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h"
Sep 11 18:18:05.036670 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.036652    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6v6g4\""
Sep 11 18:18:05.036948 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.036932    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 18:18:05.037108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.037085    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 18:18:05.049849 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.049825    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h"]
Sep 11 18:18:05.162297 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.162274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqtks\" (UniqueName: \"kubernetes.io/projected/ab1abbea-be88-42ea-a22b-4ac4669e8f40-kube-api-access-mqtks\") pod \"migrator-8b4d697c5-sjk9h\" (UID: \"ab1abbea-be88-42ea-a22b-4ac4669e8f40\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h"
Sep 11 18:18:05.263279 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.263246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqtks\" (UniqueName: \"kubernetes.io/projected/ab1abbea-be88-42ea-a22b-4ac4669e8f40-kube-api-access-mqtks\") pod \"migrator-8b4d697c5-sjk9h\" (UID: \"ab1abbea-be88-42ea-a22b-4ac4669e8f40\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h"
Sep 11 18:18:05.272261 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.272241    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqtks\" (UniqueName: \"kubernetes.io/projected/ab1abbea-be88-42ea-a22b-4ac4669e8f40-kube-api-access-mqtks\") pod \"migrator-8b4d697c5-sjk9h\" (UID: \"ab1abbea-be88-42ea-a22b-4ac4669e8f40\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h"
Sep 11 18:18:05.342652 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.342600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h"
Sep 11 18:18:05.445751 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.445727    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zw5qz"]
Sep 11 18:18:05.449016 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.449001    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.451947 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.451924    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 18:18:05.452073 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.451962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 18:18:05.452073 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.451977    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wf5gj\""
Sep 11 18:18:05.464088 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.464067    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zw5qz"]
Sep 11 18:18:05.565718 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.565693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c112901-362e-4876-9fdd-155b74b211a5-crio-socket\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.565842 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.565721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.565842 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.565757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6kls\" (UniqueName: \"kubernetes.io/projected/3c112901-362e-4876-9fdd-155b74b211a5-kube-api-access-d6kls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.565965 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.565840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c112901-362e-4876-9fdd-155b74b211a5-data-volume\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.565965 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.565911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.624290 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.624227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h"]
Sep 11 18:18:05.626748 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:05.626727    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1abbea_be88_42ea_a22b_4ac4669e8f40.slice/crio-134cd7434bdb5c06e402a6f6a38cfe751d5c66ce424635ebb2dce5ced9863eb4 WatchSource:0}: Error finding container 134cd7434bdb5c06e402a6f6a38cfe751d5c66ce424635ebb2dce5ced9863eb4: Status 404 returned error can't find the container with id 134cd7434bdb5c06e402a6f6a38cfe751d5c66ce424635ebb2dce5ced9863eb4
Sep 11 18:18:05.667234 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.667216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.667343 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.667284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c112901-362e-4876-9fdd-155b74b211a5-crio-socket\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.667343 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.667318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.667481 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:05.667353    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:05.667481 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.667397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6kls\" (UniqueName: \"kubernetes.io/projected/3c112901-362e-4876-9fdd-155b74b211a5-kube-api-access-d6kls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.667481 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:05.667418    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls podName:3c112901-362e-4876-9fdd-155b74b211a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:06.167405236 +0000 UTC m=+141.561920239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zw5qz" (UID: "3c112901-362e-4876-9fdd-155b74b211a5") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:05.667481 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.667419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c112901-362e-4876-9fdd-155b74b211a5-crio-socket\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.667481 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:05.667444    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:18:05.667481 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.667465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c112901-362e-4876-9fdd-155b74b211a5-data-volume\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.667716 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:05.667501    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm podName:3c112901-362e-4876-9fdd-155b74b211a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:06.167488233 +0000 UTC m=+141.562003237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zw5qz" (UID: "3c112901-362e-4876-9fdd-155b74b211a5") : configmap "kube-rbac-proxy" not found
Sep 11 18:18:05.667764 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.667739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c112901-362e-4876-9fdd-155b74b211a5-data-volume\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.678540 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.678514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6kls\" (UniqueName: \"kubernetes.io/projected/3c112901-362e-4876-9fdd-155b74b211a5-kube-api-access-d6kls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:05.708036 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.708014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h" event={"ID":"ab1abbea-be88-42ea-a22b-4ac4669e8f40","Type":"ContainerStarted","Data":"134cd7434bdb5c06e402a6f6a38cfe751d5c66ce424635ebb2dce5ced9863eb4"}
Sep 11 18:18:05.709344 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.709325    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/1.log"
Sep 11 18:18:05.709617 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:05.709604    2563 scope.go:117] "RemoveContainer" containerID="defc8fe71245cbc01f628864176bb63cd26487a204b7d3b2f9837ac014a226a3"
Sep 11 18:18:05.709739 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:05.709723    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-lgjg8_openshift-console-operator(fdd54bfd-280d-48a9-bdaf-4a38738c6856)\"" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" podUID="fdd54bfd-280d-48a9-bdaf-4a38738c6856"
Sep 11 18:18:06.171359 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:06.171329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:06.171772 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:06.171448    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:18:06.171772 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:06.171456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:06.171772 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:06.171511    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm podName:3c112901-362e-4876-9fdd-155b74b211a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:07.171492573 +0000 UTC m=+142.566007577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zw5qz" (UID: "3c112901-362e-4876-9fdd-155b74b211a5") : configmap "kube-rbac-proxy" not found
Sep 11 18:18:06.171772 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:06.171557    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:06.171772 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:06.171603    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls podName:3c112901-362e-4876-9fdd-155b74b211a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:07.171588345 +0000 UTC m=+142.566103350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zw5qz" (UID: "3c112901-362e-4876-9fdd-155b74b211a5") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:07.179258 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:07.179230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:07.179611 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:07.179298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:07.179611 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:07.179360    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:18:07.179611 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:07.179404    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:07.179611 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:07.179431    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm podName:3c112901-362e-4876-9fdd-155b74b211a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:09.179417721 +0000 UTC m=+144.573932724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zw5qz" (UID: "3c112901-362e-4876-9fdd-155b74b211a5") : configmap "kube-rbac-proxy" not found
Sep 11 18:18:07.179611 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:07.179445    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls podName:3c112901-362e-4876-9fdd-155b74b211a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:09.179439177 +0000 UTC m=+144.573954180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zw5qz" (UID: "3c112901-362e-4876-9fdd-155b74b211a5") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:07.717100 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:07.717074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h" event={"ID":"ab1abbea-be88-42ea-a22b-4ac4669e8f40","Type":"ContainerStarted","Data":"424561a81e5f2efdfc9aa05b16da1cf71c081d3230a2eeed47a52e9b2a29e730"}
Sep 11 18:18:07.717212 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:07.717117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h" event={"ID":"ab1abbea-be88-42ea-a22b-4ac4669e8f40","Type":"ContainerStarted","Data":"4fd33eb55785302f9b0ea54c7c0356c9f3cb099ec2f264f5f1fba72432631c90"}
Sep 11 18:18:07.734054 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:07.734020    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-sjk9h" podStartSLOduration=1.701947676 podStartE2EDuration="2.734007679s" totalImagesPullingTime="1.032060003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:07.733275636 +0000 UTC m=+143.127790674" watchObservedRunningTime="2026-09-11 18:18:07.734007679 +0000 UTC m=+143.128522702"
Sep 11 18:18:08.086337 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.086287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:08.086440 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.086336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:08.086440 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:08.086427    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 18:18:08.086508 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:08.086435    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:16.08642203 +0000 UTC m=+151.480937037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : configmap references non-existent config key: service-ca.crt
Sep 11 18:18:08.086508 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:08.086462    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:16.086451833 +0000 UTC m=+151.480966857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : secret "router-metrics-certs-default" not found
Sep 11 18:18:08.187280 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.187258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vpl2x\" (UID: \"87da72fc-2384-4c58-a47b-06ec0d24de09\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:08.187560 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:08.187400    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:18:08.187560 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:08.187446    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls podName:87da72fc-2384-4c58-a47b-06ec0d24de09 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:16.187436514 +0000 UTC m=+151.581951522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-vpl2x" (UID: "87da72fc-2384-4c58-a47b-06ec0d24de09") : secret "samples-operator-tls" not found
Sep 11 18:18:08.192129 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.192107    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qfkcq_3725edf8-bf10-4f95-99cb-2dc2581932a4/dns-node-resolver/0.log"
Sep 11 18:18:08.496015 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.495993    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-pm8xh"]
Sep 11 18:18:08.498776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.498762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-pm8xh"
Sep 11 18:18:08.501223 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.501196    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 18:18:08.501223 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.501207    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 18:18:08.501427 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.501339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xq9js\""
Sep 11 18:18:08.502407 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.502393    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 18:18:08.502477 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.502395    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 18:18:08.506722 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.506702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-pm8xh"]
Sep 11 18:18:08.589920 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.589901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/529b88a3-2b41-4f35-98a6-e8ae4ee1455c-signing-cabundle\") pod \"service-ca-568869fdc5-pm8xh\" (UID: \"529b88a3-2b41-4f35-98a6-e8ae4ee1455c\") " pod="openshift-service-ca/service-ca-568869fdc5-pm8xh"
Sep 11 18:18:08.590006 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.589985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/529b88a3-2b41-4f35-98a6-e8ae4ee1455c-signing-key\") pod \"service-ca-568869fdc5-pm8xh\" (UID: \"529b88a3-2b41-4f35-98a6-e8ae4ee1455c\") " pod="openshift-service-ca/service-ca-568869fdc5-pm8xh"
Sep 11 18:18:08.590044 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.590032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjqv\" (UniqueName: \"kubernetes.io/projected/529b88a3-2b41-4f35-98a6-e8ae4ee1455c-kube-api-access-qrjqv\") pod \"service-ca-568869fdc5-pm8xh\" (UID: \"529b88a3-2b41-4f35-98a6-e8ae4ee1455c\") " pod="openshift-service-ca/service-ca-568869fdc5-pm8xh"
Sep 11 18:18:08.691194 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.691174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/529b88a3-2b41-4f35-98a6-e8ae4ee1455c-signing-key\") pod \"service-ca-568869fdc5-pm8xh\" (UID: \"529b88a3-2b41-4f35-98a6-e8ae4ee1455c\") " pod="openshift-service-ca/service-ca-568869fdc5-pm8xh"
Sep 11 18:18:08.691276 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.691230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjqv\" (UniqueName: \"kubernetes.io/projected/529b88a3-2b41-4f35-98a6-e8ae4ee1455c-kube-api-access-qrjqv\") pod \"service-ca-568869fdc5-pm8xh\" (UID: \"529b88a3-2b41-4f35-98a6-e8ae4ee1455c\") " pod="openshift-service-ca/service-ca-568869fdc5-pm8xh"
Sep 11 18:18:08.691276 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.691260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/529b88a3-2b41-4f35-98a6-e8ae4ee1455c-signing-cabundle\") pod \"service-ca-568869fdc5-pm8xh\" (UID: \"529b88a3-2b41-4f35-98a6-e8ae4ee1455c\") " pod="openshift-service-ca/service-ca-568869fdc5-pm8xh"
Sep 11 18:18:08.691828 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.691813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/529b88a3-2b41-4f35-98a6-e8ae4ee1455c-signing-cabundle\") pod \"service-ca-568869fdc5-pm8xh\" (UID: \"529b88a3-2b41-4f35-98a6-e8ae4ee1455c\") " pod="openshift-service-ca/service-ca-568869fdc5-pm8xh"
Sep 11 18:18:08.693562 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.693543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/529b88a3-2b41-4f35-98a6-e8ae4ee1455c-signing-key\") pod \"service-ca-568869fdc5-pm8xh\" (UID: \"529b88a3-2b41-4f35-98a6-e8ae4ee1455c\") " pod="openshift-service-ca/service-ca-568869fdc5-pm8xh"
Sep 11 18:18:08.699968 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.699943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjqv\" (UniqueName: \"kubernetes.io/projected/529b88a3-2b41-4f35-98a6-e8ae4ee1455c-kube-api-access-qrjqv\") pod \"service-ca-568869fdc5-pm8xh\" (UID: \"529b88a3-2b41-4f35-98a6-e8ae4ee1455c\") " pod="openshift-service-ca/service-ca-568869fdc5-pm8xh"
Sep 11 18:18:08.808254 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:08.808215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-pm8xh"
Sep 11 18:18:09.079148 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:09.079096    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-pm8xh"]
Sep 11 18:18:09.081317 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:09.081293    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529b88a3_2b41_4f35_98a6_e8ae4ee1455c.slice/crio-8b192a035fd87391664aacda84b71a345f547751c093190480f8ae61e539ce76 WatchSource:0}: Error finding container 8b192a035fd87391664aacda84b71a345f547751c093190480f8ae61e539ce76: Status 404 returned error can't find the container with id 8b192a035fd87391664aacda84b71a345f547751c093190480f8ae61e539ce76
Sep 11 18:18:09.195487 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:09.195462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:09.195750 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:09.195515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:09.195750 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:09.195603    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:18:09.195750 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:09.195657    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm podName:3c112901-362e-4876-9fdd-155b74b211a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:13.195644314 +0000 UTC m=+148.590159317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zw5qz" (UID: "3c112901-362e-4876-9fdd-155b74b211a5") : configmap "kube-rbac-proxy" not found
Sep 11 18:18:09.195750 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:09.195603    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:09.195750 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:09.195723    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls podName:3c112901-362e-4876-9fdd-155b74b211a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:13.195711528 +0000 UTC m=+148.590226555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zw5qz" (UID: "3c112901-362e-4876-9fdd-155b74b211a5") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:09.391939 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:09.391892    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zkrtp_9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0/node-ca/0.log"
Sep 11 18:18:09.724148 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:09.724122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-pm8xh" event={"ID":"529b88a3-2b41-4f35-98a6-e8ae4ee1455c","Type":"ContainerStarted","Data":"8b192a035fd87391664aacda84b71a345f547751c093190480f8ae61e539ce76"}
Sep 11 18:18:09.996413 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:09.996336    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tw4p9_4e8d1a68-a884-44ec-b9cc-f6ad7f74156c/serve-healthcheck-canary/0.log"
Sep 11 18:18:10.394413 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:10.394335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-sjk9h_ab1abbea-be88-42ea-a22b-4ac4669e8f40/migrator/0.log"
Sep 11 18:18:10.597234 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:10.597202    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-sjk9h_ab1abbea-be88-42ea-a22b-4ac4669e8f40/graceful-termination/0.log"
Sep 11 18:18:10.873679 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:10.873649    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:10.873841 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:10.873715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:10.874130 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:10.874113    2563 scope.go:117] "RemoveContainer" containerID="defc8fe71245cbc01f628864176bb63cd26487a204b7d3b2f9837ac014a226a3"
Sep 11 18:18:10.874297 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:10.874274    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-lgjg8_openshift-console-operator(fdd54bfd-280d-48a9-bdaf-4a38738c6856)\"" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" podUID="fdd54bfd-280d-48a9-bdaf-4a38738c6856"
Sep 11 18:18:11.734484 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:11.734449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-pm8xh" event={"ID":"529b88a3-2b41-4f35-98a6-e8ae4ee1455c","Type":"ContainerStarted","Data":"f06a5d3841cc3d9bd073be76107c2aafec184fd3d0e4baa9d9b30364bcc45b80"}
Sep 11 18:18:11.756347 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:11.756295    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-pm8xh" podStartSLOduration=1.671568656 podStartE2EDuration="3.756277189s" totalImagesPullingTime="2.084708533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:11.754185507 +0000 UTC m=+147.148700543" watchObservedRunningTime="2026-09-11 18:18:11.756277189 +0000 UTC m=+147.150792216"
Sep 11 18:18:13.227659 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:13.227624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:13.228190 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:13.227743    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:13.228335 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:13.228217    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:18:13.228335 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:13.228237    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls podName:3c112901-362e-4876-9fdd-155b74b211a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:21.228217808 +0000 UTC m=+156.622732818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zw5qz" (UID: "3c112901-362e-4876-9fdd-155b74b211a5") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:13.228335 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:13.228165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:13.228335 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:13.228268    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm podName:3c112901-362e-4876-9fdd-155b74b211a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:21.22825373 +0000 UTC m=+156.622768737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zw5qz" (UID: "3c112901-362e-4876-9fdd-155b74b211a5") : configmap "kube-rbac-proxy" not found
Sep 11 18:18:16.150716 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:16.150683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:16.151061 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:16.150751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:16.151061 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:16.150848    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 18:18:16.151061 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:16.150877    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:32.150864524 +0000 UTC m=+167.545379527 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : configmap references non-existent config key: service-ca.crt
Sep 11 18:18:16.151061 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:16.150912    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs podName:4c33a960-4be6-4055-9855-b13ccf170c79 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:32.15088398 +0000 UTC m=+167.545398982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs") pod "router-default-5dbccf7754-qnh7w" (UID: "4c33a960-4be6-4055-9855-b13ccf170c79") : secret "router-metrics-certs-default" not found
Sep 11 18:18:16.251434 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:16.251407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vpl2x\" (UID: \"87da72fc-2384-4c58-a47b-06ec0d24de09\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:16.253685 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:16.253666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87da72fc-2384-4c58-a47b-06ec0d24de09-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vpl2x\" (UID: \"87da72fc-2384-4c58-a47b-06ec0d24de09\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:16.372684 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:16.372666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"
Sep 11 18:18:16.643193 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:16.643162    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x"]
Sep 11 18:18:16.754143 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:16.754100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x" event={"ID":"87da72fc-2384-4c58-a47b-06ec0d24de09","Type":"ContainerStarted","Data":"e9de7390602b10a4cc268e5a4ddb0c642270cc5291935fb2963b53c2b80e314d"}
Sep 11 18:18:18.762829 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:18.762783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x" event={"ID":"87da72fc-2384-4c58-a47b-06ec0d24de09","Type":"ContainerStarted","Data":"242a6f173d5a3024150d9f8bc636042ad44b68ad96ae7b9a3e53973692aed1eb"}
Sep 11 18:18:18.763219 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:18.762856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x" event={"ID":"87da72fc-2384-4c58-a47b-06ec0d24de09","Type":"ContainerStarted","Data":"617847e85a2066f55095e3400f60e17fa86397c4eeb3cd2a7c7deda544d79798"}
Sep 11 18:18:18.788032 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:18.787986    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vpl2x" podStartSLOduration=17.144610561 podStartE2EDuration="18.787971358s" totalImagesPullingTime="1.643360797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:18.787918358 +0000 UTC m=+154.182433382" watchObservedRunningTime="2026-09-11 18:18:18.787971358 +0000 UTC m=+154.182486381"
Sep 11 18:18:21.004611 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:21.004575    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-d672m" podUID="f4fa4956-4afa-42b9-b95f-94cb9cc018df"
Sep 11 18:18:21.120810 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:21.120779    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-tc5jj" podUID="d24dd9d4-b46c-4d50-8d1e-7b494d53cb17"
Sep 11 18:18:21.290611 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:21.290549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:21.290714 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:21.290627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:21.290714 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:21.290673    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:21.290780 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:21.290730    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls podName:3c112901-362e-4876-9fdd-155b74b211a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:37.290715854 +0000 UTC m=+172.685230863 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zw5qz" (UID: "3c112901-362e-4876-9fdd-155b74b211a5") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:21.291174 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:21.291156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c112901-362e-4876-9fdd-155b74b211a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:21.771139 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:21.771114    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d672m"
Sep 11 18:18:25.108843 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:25.108813    2563 scope.go:117] "RemoveContainer" containerID="defc8fe71245cbc01f628864176bb63cd26487a204b7d3b2f9837ac014a226a3"
Sep 11 18:18:25.784753 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:25.784731    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:18:25.785120 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:25.785107    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/1.log"
Sep 11 18:18:25.785182 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:25.785135    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdd54bfd-280d-48a9-bdaf-4a38738c6856" containerID="70749cc262e6c7d398b2d7b73c20bd465d72a03b8fe51db95dcb724469bd7aa9" exitCode=255
Sep 11 18:18:25.785214 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:25.785182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" event={"ID":"fdd54bfd-280d-48a9-bdaf-4a38738c6856","Type":"ContainerDied","Data":"70749cc262e6c7d398b2d7b73c20bd465d72a03b8fe51db95dcb724469bd7aa9"}
Sep 11 18:18:25.785243 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:25.785225    2563 scope.go:117] "RemoveContainer" containerID="defc8fe71245cbc01f628864176bb63cd26487a204b7d3b2f9837ac014a226a3"
Sep 11 18:18:25.785494 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:25.785477    2563 scope.go:117] "RemoveContainer" containerID="70749cc262e6c7d398b2d7b73c20bd465d72a03b8fe51db95dcb724469bd7aa9"
Sep 11 18:18:25.785688 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:25.785666    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-lgjg8_openshift-console-operator(fdd54bfd-280d-48a9-bdaf-4a38738c6856)\"" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" podUID="fdd54bfd-280d-48a9-bdaf-4a38738c6856"
Sep 11 18:18:25.819033 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:25.819009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:18:25.820990 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:25.820973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4fa4956-4afa-42b9-b95f-94cb9cc018df-metrics-tls\") pod \"dns-default-d672m\" (UID: \"f4fa4956-4afa-42b9-b95f-94cb9cc018df\") " pod="openshift-dns/dns-default-d672m"
Sep 11 18:18:25.974512 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:25.974489    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bf6d6\""
Sep 11 18:18:25.982856 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:25.982832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d672m"
Sep 11 18:18:26.249278 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:26.249252    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d672m"]
Sep 11 18:18:26.252229 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:26.252204    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fa4956_4afa_42b9_b95f_94cb9cc018df.slice/crio-8ae31b778a43c7b1518b39e0c05a5f562d8dba5ff314deeff815cbc956daad9b WatchSource:0}: Error finding container 8ae31b778a43c7b1518b39e0c05a5f562d8dba5ff314deeff815cbc956daad9b: Status 404 returned error can't find the container with id 8ae31b778a43c7b1518b39e0c05a5f562d8dba5ff314deeff815cbc956daad9b
Sep 11 18:18:26.790412 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:26.790354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d672m" event={"ID":"f4fa4956-4afa-42b9-b95f-94cb9cc018df","Type":"ContainerStarted","Data":"8ae31b778a43c7b1518b39e0c05a5f562d8dba5ff314deeff815cbc956daad9b"}
Sep 11 18:18:26.792003 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:26.791972    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:18:28.800443 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:28.800400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d672m" event={"ID":"f4fa4956-4afa-42b9-b95f-94cb9cc018df","Type":"ContainerStarted","Data":"0bec11959992719ed476f21fef127cdf7ee09601226aa3e666af27aef81d04c8"}
Sep 11 18:18:28.800797 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:28.800453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d672m" event={"ID":"f4fa4956-4afa-42b9-b95f-94cb9cc018df","Type":"ContainerStarted","Data":"565cd6eae85406c20ded5fced4738de92b3d9114b3331bf3882409229a67e833"}
Sep 11 18:18:28.800797 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:28.800529    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d672m"
Sep 11 18:18:28.823657 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:28.823615    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d672m" podStartSLOduration=130.360127212 podStartE2EDuration="2m11.823602409s" totalImagesPullingTime="1.463475197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:28.822849727 +0000 UTC m=+164.217364774" watchObservedRunningTime="2026-09-11 18:18:28.823602409 +0000 UTC m=+164.218117480"
Sep 11 18:18:30.874044 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:30.874015    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:30.874485 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:30.874060    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:30.874485 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:30.874315    2563 scope.go:117] "RemoveContainer" containerID="70749cc262e6c7d398b2d7b73c20bd465d72a03b8fe51db95dcb724469bd7aa9"
Sep 11 18:18:30.874485 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:30.874467    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-lgjg8_openshift-console-operator(fdd54bfd-280d-48a9-bdaf-4a38738c6856)\"" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" podUID="fdd54bfd-280d-48a9-bdaf-4a38738c6856"
Sep 11 18:18:32.163971 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:32.163944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:32.164422 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:32.164018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:32.164661 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:32.164637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c33a960-4be6-4055-9855-b13ccf170c79-service-ca-bundle\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:32.166409 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:32.166392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c33a960-4be6-4055-9855-b13ccf170c79-metrics-certs\") pod \"router-default-5dbccf7754-qnh7w\" (UID: \"4c33a960-4be6-4055-9855-b13ccf170c79\") " pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:32.169266 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:32.169253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:32.440251 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:32.440165    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5dbccf7754-qnh7w"]
Sep 11 18:18:32.444089 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:32.444065    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c33a960_4be6_4055_9855_b13ccf170c79.slice/crio-76a887306979d208c62f6065ae477c3f2d90a05fc4200dbed1f868fed7b66940 WatchSource:0}: Error finding container 76a887306979d208c62f6065ae477c3f2d90a05fc4200dbed1f868fed7b66940: Status 404 returned error can't find the container with id 76a887306979d208c62f6065ae477c3f2d90a05fc4200dbed1f868fed7b66940
Sep 11 18:18:32.815160 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:32.815075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5dbccf7754-qnh7w" event={"ID":"4c33a960-4be6-4055-9855-b13ccf170c79","Type":"ContainerStarted","Data":"7b41da2470ffdfc413048d5d0e140d6ce6843d2bef0f49dc924eec6846c32cca"}
Sep 11 18:18:32.815160 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:32.815134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5dbccf7754-qnh7w" event={"ID":"4c33a960-4be6-4055-9855-b13ccf170c79","Type":"ContainerStarted","Data":"76a887306979d208c62f6065ae477c3f2d90a05fc4200dbed1f868fed7b66940"}
Sep 11 18:18:32.844092 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:32.844045    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5dbccf7754-qnh7w" podStartSLOduration=32.844030672 podStartE2EDuration="32.844030672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:32.843395362 +0000 UTC m=+168.237910402" watchObservedRunningTime="2026-09-11 18:18:32.844030672 +0000 UTC m=+168.238545694"
Sep 11 18:18:33.169914 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:33.169895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:33.172055 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:33.172037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:33.817958 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:33.817929    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:33.818995 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:33.818978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5dbccf7754-qnh7w"
Sep 11 18:18:34.107156 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:34.107076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:18:37.297203 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:37.297173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:37.299487 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:37.299460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c112901-362e-4876-9fdd-155b74b211a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw5qz\" (UID: \"3c112901-362e-4876-9fdd-155b74b211a5\") " pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:37.558273 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:37.558217    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zw5qz"
Sep 11 18:18:37.839872 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:37.839653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zw5qz"]
Sep 11 18:18:37.842320 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:37.842295    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c112901_362e_4876_9fdd_155b74b211a5.slice/crio-0fd0f71a20e3ca4b06852a782f4f6eb2cf1d671097bf6e32bfc0970051c2863a WatchSource:0}: Error finding container 0fd0f71a20e3ca4b06852a782f4f6eb2cf1d671097bf6e32bfc0970051c2863a: Status 404 returned error can't find the container with id 0fd0f71a20e3ca4b06852a782f4f6eb2cf1d671097bf6e32bfc0970051c2863a
Sep 11 18:18:38.805977 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:38.805958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d672m"
Sep 11 18:18:38.835355 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:38.835287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zw5qz" event={"ID":"3c112901-362e-4876-9fdd-155b74b211a5","Type":"ContainerStarted","Data":"f02cecfcef9c28c701087162503a1231b9d2aaab48b5d274f74dcc49f53094c3"}
Sep 11 18:18:38.835355 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:38.835335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zw5qz" event={"ID":"3c112901-362e-4876-9fdd-155b74b211a5","Type":"ContainerStarted","Data":"6df220587b157d6e8692311407807f96da11d199461609cc49dec2e991690782"}
Sep 11 18:18:38.835542 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:38.835361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zw5qz" event={"ID":"3c112901-362e-4876-9fdd-155b74b211a5","Type":"ContainerStarted","Data":"0fd0f71a20e3ca4b06852a782f4f6eb2cf1d671097bf6e32bfc0970051c2863a"}
Sep 11 18:18:40.844260 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:40.844231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zw5qz" event={"ID":"3c112901-362e-4876-9fdd-155b74b211a5","Type":"ContainerStarted","Data":"4fbbaee5a827e65fa1d747cd791436bfb20adb233e779ffe4185370447226911"}
Sep 11 18:18:40.868809 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:40.868764    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zw5qz" podStartSLOduration=34.02822439 podStartE2EDuration="35.868749969s" totalImagesPullingTime="1.840525579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:18:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:40.867007413 +0000 UTC m=+176.261522449" watchObservedRunningTime="2026-09-11 18:18:40.868749969 +0000 UTC m=+176.263264997"
Sep 11 18:18:41.107350 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.107287    2563 scope.go:117] "RemoveContainer" containerID="70749cc262e6c7d398b2d7b73c20bd465d72a03b8fe51db95dcb724469bd7aa9"
Sep 11 18:18:41.107494 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:18:41.107463    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-lgjg8_openshift-console-operator(fdd54bfd-280d-48a9-bdaf-4a38738c6856)\"" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" podUID="fdd54bfd-280d-48a9-bdaf-4a38738c6856"
Sep 11 18:18:41.343635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.343607    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pbjjb"]
Sep 11 18:18:41.346784 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.346768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.349330 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.349308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 18:18:41.349467 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.349355    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 18:18:41.349551 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.349529    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 18:18:41.349879 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.349843    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 18:18:41.349879 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.349865    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 18:18:41.350209 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.350194    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-962fk\""
Sep 11 18:18:41.350519 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.350500    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 18:18:41.527623 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.527598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-wtmp\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.527721 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.527634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vs6z\" (UniqueName: \"kubernetes.io/projected/ef955b9e-81c8-4611-baa6-29c300923c7f-kube-api-access-2vs6z\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.527721 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.527658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-tls\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.527788 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.527723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.527788 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.527746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef955b9e-81c8-4611-baa6-29c300923c7f-root\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.527788 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.527765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-textfile\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.527879 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.527824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef955b9e-81c8-4611-baa6-29c300923c7f-metrics-client-ca\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.527879 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.527867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef955b9e-81c8-4611-baa6-29c300923c7f-sys\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.527937 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.527896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628109 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef955b9e-81c8-4611-baa6-29c300923c7f-metrics-client-ca\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628206 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef955b9e-81c8-4611-baa6-29c300923c7f-sys\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628206 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628206 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-wtmp\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef955b9e-81c8-4611-baa6-29c300923c7f-sys\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vs6z\" (UniqueName: \"kubernetes.io/projected/ef955b9e-81c8-4611-baa6-29c300923c7f-kube-api-access-2vs6z\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-tls\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-wtmp\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628580 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef955b9e-81c8-4611-baa6-29c300923c7f-root\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628580 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-textfile\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628580 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef955b9e-81c8-4611-baa6-29c300923c7f-root\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628751 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-textfile\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628802 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef955b9e-81c8-4611-baa6-29c300923c7f-metrics-client-ca\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.628850 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.628779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.630447 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.630427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.630587 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.630567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef955b9e-81c8-4611-baa6-29c300923c7f-node-exporter-tls\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.638988 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.638963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vs6z\" (UniqueName: \"kubernetes.io/projected/ef955b9e-81c8-4611-baa6-29c300923c7f-kube-api-access-2vs6z\") pod \"node-exporter-pbjjb\" (UID: \"ef955b9e-81c8-4611-baa6-29c300923c7f\") " pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.657363 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.657340    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pbjjb"
Sep 11 18:18:41.673159 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:41.673138    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef955b9e_81c8_4611_baa6_29c300923c7f.slice/crio-755d47582ff036fad0a1215eafd18bdc8829b91999a62a81fa467006e6c1fc87 WatchSource:0}: Error finding container 755d47582ff036fad0a1215eafd18bdc8829b91999a62a81fa467006e6c1fc87: Status 404 returned error can't find the container with id 755d47582ff036fad0a1215eafd18bdc8829b91999a62a81fa467006e6c1fc87
Sep 11 18:18:41.848335 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:41.848278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbjjb" event={"ID":"ef955b9e-81c8-4611-baa6-29c300923c7f","Type":"ContainerStarted","Data":"755d47582ff036fad0a1215eafd18bdc8829b91999a62a81fa467006e6c1fc87"}
Sep 11 18:18:42.853614 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:42.853538    2563 generic.go:356] "Generic (PLEG): container finished" podID="ef955b9e-81c8-4611-baa6-29c300923c7f" containerID="42cc9bbd06227e81821e47b286c0899d3ad3698b147557ad2cec84cd7985510b" exitCode=0
Sep 11 18:18:42.853960 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:42.853613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbjjb" event={"ID":"ef955b9e-81c8-4611-baa6-29c300923c7f","Type":"ContainerDied","Data":"42cc9bbd06227e81821e47b286c0899d3ad3698b147557ad2cec84cd7985510b"}
Sep 11 18:18:43.862649 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:43.862610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbjjb" event={"ID":"ef955b9e-81c8-4611-baa6-29c300923c7f","Type":"ContainerStarted","Data":"5879e11c96f97013782346116f4f1bb06f295c4d273fa37abf68bb0832ce092b"}
Sep 11 18:18:43.863109 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:43.862699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbjjb" event={"ID":"ef955b9e-81c8-4611-baa6-29c300923c7f","Type":"ContainerStarted","Data":"f8f1d88610f8508ebfc813717c4b961471458f5d1f6cf31ccf83e23e3f201db7"}
Sep 11 18:18:43.885074 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:43.885026    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pbjjb" podStartSLOduration=2.030901412 podStartE2EDuration="2.885011959s" totalImagesPullingTime="795.324122ms" totalInitContainerRuntime="58.786425ms" isStatefulPod=true podCreationTimestamp="2026-09-11 18:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:43.88339515 +0000 UTC m=+179.277910179" watchObservedRunningTime="2026-09-11 18:18:43.885011959 +0000 UTC m=+179.279526983"
Sep 11 18:18:44.300019 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.299993    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"]
Sep 11 18:18:44.304815 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.304800    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.307797 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.307779    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 18:18:44.307950 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.307935    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c5p83er188jmo\""
Sep 11 18:18:44.308047 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.308002    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 18:18:44.308116 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.308099    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 18:18:44.308274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.308262    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 18:18:44.308880 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.308866    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-h8wcp\""
Sep 11 18:18:44.308919 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.308894    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 18:18:44.318025 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.318007    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"]
Sep 11 18:18:44.448035 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.448009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87597028-7fc4-4cdd-aeb2-317648c732c0-metrics-client-ca\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.448186 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.448045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-tls\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.448186 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.448063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.448186 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.448141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scw5k\" (UniqueName: \"kubernetes.io/projected/87597028-7fc4-4cdd-aeb2-317648c732c0-kube-api-access-scw5k\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.448347 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.448205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.448347 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.448226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.448347 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.448256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.448347 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.448294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-grpc-tls\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.548579 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.548553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scw5k\" (UniqueName: \"kubernetes.io/projected/87597028-7fc4-4cdd-aeb2-317648c732c0-kube-api-access-scw5k\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.548681 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.548609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.548681 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.548639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.548784 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.548679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.548784 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.548714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-grpc-tls\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.548894 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.548872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87597028-7fc4-4cdd-aeb2-317648c732c0-metrics-client-ca\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.548939 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.548927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-tls\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.548982 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.548952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.549713 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.549688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87597028-7fc4-4cdd-aeb2-317648c732c0-metrics-client-ca\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.552168 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.552095    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-grpc-tls\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.552479 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.552438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.552479 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.552465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-tls\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.552644 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.552579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.552699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.552640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.552699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.552663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87597028-7fc4-4cdd-aeb2-317648c732c0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.562107 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.562083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scw5k\" (UniqueName: \"kubernetes.io/projected/87597028-7fc4-4cdd-aeb2-317648c732c0-kube-api-access-scw5k\") pod \"thanos-querier-79446bcfd8-5xjmk\" (UID: \"87597028-7fc4-4cdd-aeb2-317648c732c0\") " pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.613435 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.613391    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:44.894134 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:44.894117    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"]
Sep 11 18:18:44.895916 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:44.895891    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87597028_7fc4_4cdd_aeb2_317648c732c0.slice/crio-f456ad9d2163094c4ecaae3d1df2794b0baaf6969f33ba35a7d038a25f53ae8b WatchSource:0}: Error finding container f456ad9d2163094c4ecaae3d1df2794b0baaf6969f33ba35a7d038a25f53ae8b: Status 404 returned error can't find the container with id f456ad9d2163094c4ecaae3d1df2794b0baaf6969f33ba35a7d038a25f53ae8b
Sep 11 18:18:45.750203 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.750162    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"]
Sep 11 18:18:45.754390 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.754352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.761328 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.761280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 18:18:45.761647 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.761617    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 18:18:45.763162 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.763138    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 18:18:45.763444 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.763394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-45ur4ih76serm\""
Sep 11 18:18:45.764014 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.763943    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nf47n\""
Sep 11 18:18:45.764115 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.764069    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 18:18:45.771031 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.771008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"]
Sep 11 18:18:45.859676 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.859645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhjfd\" (UniqueName: \"kubernetes.io/projected/7370ff00-1076-43b2-a222-ed33ef415994-kube-api-access-jhjfd\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.859799 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.859694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7370ff00-1076-43b2-a222-ed33ef415994-metrics-server-audit-profiles\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.859799 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.859751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7370ff00-1076-43b2-a222-ed33ef415994-client-ca-bundle\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.859885 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.859813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7370ff00-1076-43b2-a222-ed33ef415994-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.859885 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.859841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7370ff00-1076-43b2-a222-ed33ef415994-secret-metrics-server-tls\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.860027 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.859890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7370ff00-1076-43b2-a222-ed33ef415994-secret-metrics-server-client-certs\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.860027 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.859917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7370ff00-1076-43b2-a222-ed33ef415994-audit-log\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.871549 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.871516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk" event={"ID":"87597028-7fc4-4cdd-aeb2-317648c732c0","Type":"ContainerStarted","Data":"f456ad9d2163094c4ecaae3d1df2794b0baaf6969f33ba35a7d038a25f53ae8b"}
Sep 11 18:18:45.960168 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.960141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7370ff00-1076-43b2-a222-ed33ef415994-secret-metrics-server-client-certs\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.960484 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.960168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7370ff00-1076-43b2-a222-ed33ef415994-audit-log\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.960484 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.960189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhjfd\" (UniqueName: \"kubernetes.io/projected/7370ff00-1076-43b2-a222-ed33ef415994-kube-api-access-jhjfd\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.960484 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.960218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7370ff00-1076-43b2-a222-ed33ef415994-metrics-server-audit-profiles\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.960484 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.960430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7370ff00-1076-43b2-a222-ed33ef415994-client-ca-bundle\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.960678 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.960482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7370ff00-1076-43b2-a222-ed33ef415994-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.960678 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.960515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7370ff00-1076-43b2-a222-ed33ef415994-secret-metrics-server-tls\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.960678 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.960572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7370ff00-1076-43b2-a222-ed33ef415994-audit-log\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.961223 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.961181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7370ff00-1076-43b2-a222-ed33ef415994-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.961223 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.961185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7370ff00-1076-43b2-a222-ed33ef415994-metrics-server-audit-profiles\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.963330 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.963308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7370ff00-1076-43b2-a222-ed33ef415994-client-ca-bundle\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.963330 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.963319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7370ff00-1076-43b2-a222-ed33ef415994-secret-metrics-server-tls\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.963527 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.963444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7370ff00-1076-43b2-a222-ed33ef415994-secret-metrics-server-client-certs\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:45.970191 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:45.970163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhjfd\" (UniqueName: \"kubernetes.io/projected/7370ff00-1076-43b2-a222-ed33ef415994-kube-api-access-jhjfd\") pod \"metrics-server-59dbc9dffc-rmjfk\" (UID: \"7370ff00-1076-43b2-a222-ed33ef415994\") " pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:46.065930 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:46.065856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:18:46.791103 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:46.791030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"]
Sep 11 18:18:46.793487 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:46.793460    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7370ff00_1076_43b2_a222_ed33ef415994.slice/crio-f4882b267a2a467732a53ffe888c04581d23d26b22295bb9f810a43524989345 WatchSource:0}: Error finding container f4882b267a2a467732a53ffe888c04581d23d26b22295bb9f810a43524989345: Status 404 returned error can't find the container with id f4882b267a2a467732a53ffe888c04581d23d26b22295bb9f810a43524989345
Sep 11 18:18:46.876102 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:46.876075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk" event={"ID":"7370ff00-1076-43b2-a222-ed33ef415994","Type":"ContainerStarted","Data":"f4882b267a2a467732a53ffe888c04581d23d26b22295bb9f810a43524989345"}
Sep 11 18:18:46.877949 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:46.877931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk" event={"ID":"87597028-7fc4-4cdd-aeb2-317648c732c0","Type":"ContainerStarted","Data":"629d2ebd7be36f8f964b7755b6f41507fbd2132cafaeb2829b8f7385e4631081"}
Sep 11 18:18:46.878035 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:46.877986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk" event={"ID":"87597028-7fc4-4cdd-aeb2-317648c732c0","Type":"ContainerStarted","Data":"584f4b3e1ef5ccb0d9c112cf00c77232bcc3de9c393b29de25f6531aec361b54"}
Sep 11 18:18:46.878035 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:46.878013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk" event={"ID":"87597028-7fc4-4cdd-aeb2-317648c732c0","Type":"ContainerStarted","Data":"4a2ab81b565e8c97d031386ceb1066d4b35ac5d2b67a5340997a6a69ad4a07f4"}
Sep 11 18:18:47.688719 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.688685    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:18:47.692621 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.692600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.698286 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.698245    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 18:18:47.698286 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.698248    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 18:18:47.698540 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.698323    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 18:18:47.698540 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.698464    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 18:18:47.698679 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.698626    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 18:18:47.699101 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.698829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 18:18:47.699101 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.698857    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 18:18:47.699101 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.698908    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 18:18:47.699101 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.698862    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 18:18:47.699101 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.699022    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 18:18:47.699101 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.699101    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n4qp4\""
Sep 11 18:18:47.699466 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.699155    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 18:18:47.699466 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.699348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 18:18:47.699623 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.699607    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4c6u3d2jpudhh\""
Sep 11 18:18:47.704699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.704677    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 18:18:47.704799 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.704713    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 18:18:47.717506 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.717481    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:18:47.878103 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878479 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878479 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878479 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878479 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878644 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878644 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878644 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzcxd\" (UniqueName: \"kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-kube-api-access-dzcxd\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878644 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878644 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878877 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878877 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878877 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878877 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878877 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.878877 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.879114 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.878903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979514 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979619 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979619 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979619 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979792 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979792 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979792 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979792 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979979 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979979 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979979 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979979 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979979 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979979 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.979979 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.980301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.979984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.980301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.980022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcxd\" (UniqueName: \"kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-kube-api-access-dzcxd\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.980301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.980054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.980301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.980101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.980301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.980158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.980645 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.980609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.981563 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.981539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.983832 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.983809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.983934 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.983865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.984264 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.984245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.985889 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.985860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.987109 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.987084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.988282 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.988257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.989130 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.989102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.989229 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.989161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.989229 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.989173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.990124 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.989796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.990124 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.989909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.991275 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.990450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.991275 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.990919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.992900 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.991858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.992900 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.991888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.992900 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.992102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.994757 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.994733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:47.997166 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:47.997149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzcxd\" (UniqueName: \"kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-kube-api-access-dzcxd\") pod \"prometheus-k8s-0\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:48.004225 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:48.004201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:48.307259 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:18:48.307212    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa5d02b_083f_445d_ac2b_cecca808874c.slice/crio-92cceaab6009cf2003634743aa3305601c389f3eb66aad34075e32fbbf9472d5 WatchSource:0}: Error finding container 92cceaab6009cf2003634743aa3305601c389f3eb66aad34075e32fbbf9472d5: Status 404 returned error can't find the container with id 92cceaab6009cf2003634743aa3305601c389f3eb66aad34075e32fbbf9472d5
Sep 11 18:18:48.308242 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:48.308219    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:18:48.890007 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:48.889963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk" event={"ID":"87597028-7fc4-4cdd-aeb2-317648c732c0","Type":"ContainerStarted","Data":"a69b421ce208848b31ed130d47285e6a4b50e529e2d2da3a0a854380a351931f"}
Sep 11 18:18:48.890437 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:48.890054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk" event={"ID":"87597028-7fc4-4cdd-aeb2-317648c732c0","Type":"ContainerStarted","Data":"1ad71e9c5f3c476d07f604be575a4c8393c56accafac58d66bee2c4c678172a7"}
Sep 11 18:18:48.890437 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:48.890097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk" event={"ID":"87597028-7fc4-4cdd-aeb2-317648c732c0","Type":"ContainerStarted","Data":"58032ea1d82af4843ec8f71e1d51dff6dcc1fd025914ba698a5ff908cf7b9042"}
Sep 11 18:18:48.890437 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:48.890154    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:48.892105 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:48.892077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk" event={"ID":"7370ff00-1076-43b2-a222-ed33ef415994","Type":"ContainerStarted","Data":"f7376733321c3e1fcb02108da3cfefd692db80705018e49f4430d90f36e759c7"}
Sep 11 18:18:48.893553 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:48.893532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerStarted","Data":"92cceaab6009cf2003634743aa3305601c389f3eb66aad34075e32fbbf9472d5"}
Sep 11 18:18:48.917069 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:48.917021    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk" podStartSLOduration=2.057735188 podStartE2EDuration="4.917005653s" totalImagesPullingTime="2.859270465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:48.915881067 +0000 UTC m=+184.310396117" watchObservedRunningTime="2026-09-11 18:18:48.917005653 +0000 UTC m=+184.311520678"
Sep 11 18:18:48.936143 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:48.936099    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk" podStartSLOduration=2.7875505990000002 podStartE2EDuration="3.936084622s" totalImagesPullingTime="1.148534023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:48.935733313 +0000 UTC m=+184.330248363" watchObservedRunningTime="2026-09-11 18:18:48.936084622 +0000 UTC m=+184.330599647"
Sep 11 18:18:49.898615 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:49.898580    2563 generic.go:356] "Generic (PLEG): container finished" podID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerID="1e93f0fe1f3f452ca1807acc8bf6d51579236b2ebfa709717e30ed8b7afb121c" exitCode=0
Sep 11 18:18:49.898973 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:49.898664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerDied","Data":"1e93f0fe1f3f452ca1807acc8bf6d51579236b2ebfa709717e30ed8b7afb121c"}
Sep 11 18:18:52.913542 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:52.913517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerStarted","Data":"0afcef759c3066653e988f46b43b563c86f748b3fbbc623bb46be4199514d193"}
Sep 11 18:18:52.913827 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:52.913580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerStarted","Data":"84a4c782e08061aa769412f37e1175c0d62ed280e37b37aedec77fcd4a28cca2"}
Sep 11 18:18:52.913827 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:52.913606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerStarted","Data":"88470d7bacc41a1dfed0dcb2c14339ce55f69001d23af3bc2c4e8167723d4e49"}
Sep 11 18:18:53.920559 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:53.920527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerStarted","Data":"9645763a7e5c0bb5cb8c95c76c570021ecc74c88e4a29e5ae6eba3cab13dfd54"}
Sep 11 18:18:53.920911 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:53.920590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerStarted","Data":"eb868ba70506eb642604461f9955be7aa7d4839db2df6a82eb3e9a3b68b0bceb"}
Sep 11 18:18:53.920911 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:53.920618    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerStarted","Data":"d3c6b6290bcc1c49b9d1de2ef95c46bf2856adbfe7f86e1fc124db23e39961c3"}
Sep 11 18:18:53.957747 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:53.957702    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.143111164 podStartE2EDuration="6.957684281s" totalImagesPullingTime="3.79794997s" totalInitContainerRuntime="16.623147ms" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:53.955062147 +0000 UTC m=+189.349577185" watchObservedRunningTime="2026-09-11 18:18:53.957684281 +0000 UTC m=+189.352199306"
Sep 11 18:18:54.107088 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:54.107067    2563 scope.go:117] "RemoveContainer" containerID="70749cc262e6c7d398b2d7b73c20bd465d72a03b8fe51db95dcb724469bd7aa9"
Sep 11 18:18:54.904396 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:54.904359    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79446bcfd8-5xjmk"
Sep 11 18:18:54.926344 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:54.926325    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:18:54.926705 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:54.926455    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" event={"ID":"fdd54bfd-280d-48a9-bdaf-4a38738c6856","Type":"ContainerStarted","Data":"e4aaa334209de4148ff2d1f4e8008b00d83b8582966f2ba7de1ec005744fd34f"}
Sep 11 18:18:54.926977 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:54.926957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:54.957630 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:54.957581    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8" podStartSLOduration=52.600256763 podStartE2EDuration="54.957568819s" totalImagesPullingTime="2.357312056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:54.956364093 +0000 UTC m=+190.350879117" watchObservedRunningTime="2026-09-11 18:18:54.957568819 +0000 UTC m=+190.352083842"
Sep 11 18:18:55.313728 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:55.313654    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-lgjg8"
Sep 11 18:18:58.004949 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:18:58.004917    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:19:06.066762 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:06.066725    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:19:06.067182 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:06.066821    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:19:09.984812 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:09.984781    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7d9182c-e4a7-4850-b8f9-a43199335b21" containerID="ca32338f46db50ebe69689bced8171ed5b6192883eae8179a119c975d98059ab" exitCode=0
Sep 11 18:19:09.985121 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:09.984837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-bmtcc" event={"ID":"f7d9182c-e4a7-4850-b8f9-a43199335b21","Type":"ContainerDied","Data":"ca32338f46db50ebe69689bced8171ed5b6192883eae8179a119c975d98059ab"}
Sep 11 18:19:09.985121 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:09.985107    2563 scope.go:117] "RemoveContainer" containerID="ca32338f46db50ebe69689bced8171ed5b6192883eae8179a119c975d98059ab"
Sep 11 18:19:10.992522 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:10.992470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-bmtcc" event={"ID":"f7d9182c-e4a7-4850-b8f9-a43199335b21","Type":"ContainerStarted","Data":"b47d776c4d74bb1baef9acf5ea0ae9f94d1d307216cd3e4f5bf968611abf15ce"}
Sep 11 18:19:14.181306 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:14.181277    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-59dbc9dffc-rmjfk_7370ff00-1076-43b2-a222-ed33ef415994/metrics-server/0.log"
Sep 11 18:19:15.776672 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:15.776645    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbjjb_ef955b9e-81c8-4611-baa6-29c300923c7f/init-textfile/0.log"
Sep 11 18:19:15.977564 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:15.977537    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbjjb_ef955b9e-81c8-4611-baa6-29c300923c7f/node-exporter/0.log"
Sep 11 18:19:16.176738 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:16.176718    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbjjb_ef955b9e-81c8-4611-baa6-29c300923c7f/kube-rbac-proxy/0.log"
Sep 11 18:19:16.981618 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:16.981568    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa5d02b-083f-445d-ac2b-cecca808874c/init-config-reloader/0.log"
Sep 11 18:19:17.190125 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:17.190092    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa5d02b-083f-445d-ac2b-cecca808874c/prometheus/0.log"
Sep 11 18:19:17.389583 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:17.389493    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa5d02b-083f-445d-ac2b-cecca808874c/config-reloader/0.log"
Sep 11 18:19:17.577232 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:17.577203    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa5d02b-083f-445d-ac2b-cecca808874c/thanos-sidecar/0.log"
Sep 11 18:19:17.780723 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:17.780679    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa5d02b-083f-445d-ac2b-cecca808874c/kube-rbac-proxy-web/0.log"
Sep 11 18:19:17.978623 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:17.978591    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa5d02b-083f-445d-ac2b-cecca808874c/kube-rbac-proxy/0.log"
Sep 11 18:19:18.177697 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:18.177677    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa5d02b-083f-445d-ac2b-cecca808874c/kube-rbac-proxy-thanos/0.log"
Sep 11 18:19:19.576923 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:19.576895    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/thanos-query/0.log"
Sep 11 18:19:19.781965 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:19.781942    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/kube-rbac-proxy-web/0.log"
Sep 11 18:19:19.976495 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:19.976474    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/kube-rbac-proxy/0.log"
Sep 11 18:19:20.176911 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:20.176886    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/prom-label-proxy/0.log"
Sep 11 18:19:20.377320 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:20.377248    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/kube-rbac-proxy-rules/0.log"
Sep 11 18:19:20.583162 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:20.583136    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/kube-rbac-proxy-metrics/0.log"
Sep 11 18:19:20.978962 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:20.978934    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:19:21.178860 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:21.178837    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/3.log"
Sep 11 18:19:21.778859 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:21.778828    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dbccf7754-qnh7w_4c33a960-4be6-4055-9855-b13ccf170c79/router/0.log"
Sep 11 18:19:22.176995 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:22.176972    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tw4p9_4e8d1a68-a884-44ec-b9cc-f6ad7f74156c/serve-healthcheck-canary/0.log"
Sep 11 18:19:22.578784 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:22.578715    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-86499c89c4-vpl2x_87da72fc-2384-4c58-a47b-06ec0d24de09/cluster-samples-operator/0.log"
Sep 11 18:19:22.779266 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:22.779241    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-86499c89c4-vpl2x_87da72fc-2384-4c58-a47b-06ec0d24de09/cluster-samples-operator-watch/0.log"
Sep 11 18:19:26.078028 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:26.078000    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:19:26.083282 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:26.083258    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-59dbc9dffc-rmjfk"
Sep 11 18:19:48.005029 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:48.004995    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:19:48.020308 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:48.020286    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:19:48.153425 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:48.153403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:19:55.983951 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:55.983917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:19:55.986234 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:55.986206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24dd9d4-b46c-4d50-8d1e-7b494d53cb17-metrics-certs\") pod \"network-metrics-daemon-tc5jj\" (UID: \"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17\") " pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:19:56.010586 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:56.010560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bn27r\""
Sep 11 18:19:56.017916 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:56.017894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5jj"
Sep 11 18:19:56.289617 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:56.289486    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc5jj"]
Sep 11 18:19:56.292058 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:19:56.292030    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24dd9d4_b46c_4d50_8d1e_7b494d53cb17.slice/crio-d7164c7de38607da233e248094c9cc3bbc0efaf70bcee94ce6db0393d4d97aa7 WatchSource:0}: Error finding container d7164c7de38607da233e248094c9cc3bbc0efaf70bcee94ce6db0393d4d97aa7: Status 404 returned error can't find the container with id d7164c7de38607da233e248094c9cc3bbc0efaf70bcee94ce6db0393d4d97aa7
Sep 11 18:19:57.170163 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:57.170121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc5jj" event={"ID":"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17","Type":"ContainerStarted","Data":"d7164c7de38607da233e248094c9cc3bbc0efaf70bcee94ce6db0393d4d97aa7"}
Sep 11 18:19:58.175091 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:58.175054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc5jj" event={"ID":"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17","Type":"ContainerStarted","Data":"bce1af86cbbb5f93a0f25d8a958fb60441b64376b59f6900c3544f1ed6f89947"}
Sep 11 18:19:58.175443 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:58.175121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc5jj" event={"ID":"d24dd9d4-b46c-4d50-8d1e-7b494d53cb17","Type":"ContainerStarted","Data":"b320268e2e26b7837d79920540fe096469b7055f6ea405754d88ce5c470364aa"}
Sep 11 18:19:58.194627 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:19:58.194587    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tc5jj" podStartSLOduration=252.24300821 podStartE2EDuration="4m13.194572911s" totalImagesPullingTime="951.564701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:19:58.19331395 +0000 UTC m=+253.587828993" watchObservedRunningTime="2026-09-11 18:19:58.194572911 +0000 UTC m=+253.589087936"
Sep 11 18:20:06.081879 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.081847    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:06.082274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.082107    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="prometheus" containerID="cri-o://88470d7bacc41a1dfed0dcb2c14339ce55f69001d23af3bc2c4e8167723d4e49" gracePeriod=600
Sep 11 18:20:06.082274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.082132    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy" containerID="cri-o://eb868ba70506eb642604461f9955be7aa7d4839db2df6a82eb3e9a3b68b0bceb" gracePeriod=600
Sep 11 18:20:06.082274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.082154    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="thanos-sidecar" containerID="cri-o://0afcef759c3066653e988f46b43b563c86f748b3fbbc623bb46be4199514d193" gracePeriod=600
Sep 11 18:20:06.082274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.082182    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy-web" containerID="cri-o://d3c6b6290bcc1c49b9d1de2ef95c46bf2856adbfe7f86e1fc124db23e39961c3" gracePeriod=600
Sep 11 18:20:06.082274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.082188    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9645763a7e5c0bb5cb8c95c76c570021ecc74c88e4a29e5ae6eba3cab13dfd54" gracePeriod=600
Sep 11 18:20:06.082274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.082231    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="config-reloader" containerID="cri-o://84a4c782e08061aa769412f37e1175c0d62ed280e37b37aedec77fcd4a28cca2" gracePeriod=600
Sep 11 18:20:06.206007 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.205984    2563 generic.go:356] "Generic (PLEG): container finished" podID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerID="9645763a7e5c0bb5cb8c95c76c570021ecc74c88e4a29e5ae6eba3cab13dfd54" exitCode=0
Sep 11 18:20:06.206007 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.206005    2563 generic.go:356] "Generic (PLEG): container finished" podID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerID="eb868ba70506eb642604461f9955be7aa7d4839db2df6a82eb3e9a3b68b0bceb" exitCode=0
Sep 11 18:20:06.206121 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.206011    2563 generic.go:356] "Generic (PLEG): container finished" podID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerID="d3c6b6290bcc1c49b9d1de2ef95c46bf2856adbfe7f86e1fc124db23e39961c3" exitCode=0
Sep 11 18:20:06.206121 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.206017    2563 generic.go:356] "Generic (PLEG): container finished" podID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerID="0afcef759c3066653e988f46b43b563c86f748b3fbbc623bb46be4199514d193" exitCode=0
Sep 11 18:20:06.206121 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.206022    2563 generic.go:356] "Generic (PLEG): container finished" podID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerID="84a4c782e08061aa769412f37e1175c0d62ed280e37b37aedec77fcd4a28cca2" exitCode=0
Sep 11 18:20:06.206121 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.206029    2563 generic.go:356] "Generic (PLEG): container finished" podID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerID="88470d7bacc41a1dfed0dcb2c14339ce55f69001d23af3bc2c4e8167723d4e49" exitCode=0
Sep 11 18:20:06.206121 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.206052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerDied","Data":"9645763a7e5c0bb5cb8c95c76c570021ecc74c88e4a29e5ae6eba3cab13dfd54"}
Sep 11 18:20:06.206121 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.206114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerDied","Data":"eb868ba70506eb642604461f9955be7aa7d4839db2df6a82eb3e9a3b68b0bceb"}
Sep 11 18:20:06.206286 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.206141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerDied","Data":"d3c6b6290bcc1c49b9d1de2ef95c46bf2856adbfe7f86e1fc124db23e39961c3"}
Sep 11 18:20:06.206286 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.206169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerDied","Data":"0afcef759c3066653e988f46b43b563c86f748b3fbbc623bb46be4199514d193"}
Sep 11 18:20:06.206286 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.206194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerDied","Data":"84a4c782e08061aa769412f37e1175c0d62ed280e37b37aedec77fcd4a28cca2"}
Sep 11 18:20:06.206286 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.206216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerDied","Data":"88470d7bacc41a1dfed0dcb2c14339ce55f69001d23af3bc2c4e8167723d4e49"}
Sep 11 18:20:06.321220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.321199    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:06.456819 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.456784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-0\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.456819 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.456825    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-kubelet-serving-ca-bundle\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457018 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.456848    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-metrics-client-certs\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457018 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.456882    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-1\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457018 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.456899    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-db\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457018 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.456916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-config-out\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457018 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.456948    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-kube-rbac-proxy\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457018 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.456972    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457018 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.456998    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-grpc-tls\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457018 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457020    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-tls-assets\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-tls\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-thanos-prometheus-http-client-file\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457093    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-web-config\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457125    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-2\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457159    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-trusted-ca-bundle\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457183    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-config\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-metrics-client-ca\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457244    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzcxd\" (UniqueName: \"kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-kube-api-access-dzcxd\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457289    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:06.457366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-serving-certs-ca-bundle\") pod \"8aa5d02b-083f-445d-ac2b-cecca808874c\" (UID: \"8aa5d02b-083f-445d-ac2b-cecca808874c\") "
Sep 11 18:20:06.457905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457423    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:06.457905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457599    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.457905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457618    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.457905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.457825    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:06.458542 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.458225    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:06.458542 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.458338    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:20:06.459118 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.458876    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:06.459668 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.459636    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:06.459748 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.459662    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:06.460407 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.460095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:06.460407 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.460209    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-config-out" (OuterVolumeSpecName: "config-out") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:20:06.460407 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.460392    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:06.460921 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.460894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:06.461258 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.461233    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:06.461724 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.461694    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-kube-api-access-dzcxd" (OuterVolumeSpecName: "kube-api-access-dzcxd") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "kube-api-access-dzcxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:20:06.461724 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.461701    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-config" (OuterVolumeSpecName: "config") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:06.461865 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.461851    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:06.461979 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.461964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:20:06.462102 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.462086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:06.462206 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.462184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:06.471546 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.471525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-web-config" (OuterVolumeSpecName: "web-config") pod "8aa5d02b-083f-445d-ac2b-cecca808874c" (UID: "8aa5d02b-083f-445d-ac2b-cecca808874c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:06.558742 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558721    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558742 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558741    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558752    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-config\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558762    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-metrics-client-ca\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558779    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzcxd\" (UniqueName: \"kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-kube-api-access-dzcxd\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558788    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558797    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558805    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558813    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-metrics-client-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558822    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-prometheus-k8s-db\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558830    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5d02b-083f-445d-ac2b-cecca808874c-config-out\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558838    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-kube-rbac-proxy\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558845    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.558853 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558854    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-grpc-tls\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.559192 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558862    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5d02b-083f-445d-ac2b-cecca808874c-tls-assets\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.559192 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558869    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.559192 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558877    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:06.559192 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:06.558886    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5d02b-083f-445d-ac2b-cecca808874c-web-config\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:20:07.212912 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.212885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5d02b-083f-445d-ac2b-cecca808874c","Type":"ContainerDied","Data":"92cceaab6009cf2003634743aa3305601c389f3eb66aad34075e32fbbf9472d5"}
Sep 11 18:20:07.213217 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.212914    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.213217 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.212954    2563 scope.go:117] "RemoveContainer" containerID="9645763a7e5c0bb5cb8c95c76c570021ecc74c88e4a29e5ae6eba3cab13dfd54"
Sep 11 18:20:07.220283 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.220263    2563 scope.go:117] "RemoveContainer" containerID="eb868ba70506eb642604461f9955be7aa7d4839db2df6a82eb3e9a3b68b0bceb"
Sep 11 18:20:07.227329 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.227310    2563 scope.go:117] "RemoveContainer" containerID="d3c6b6290bcc1c49b9d1de2ef95c46bf2856adbfe7f86e1fc124db23e39961c3"
Sep 11 18:20:07.233466 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.233447    2563 scope.go:117] "RemoveContainer" containerID="0afcef759c3066653e988f46b43b563c86f748b3fbbc623bb46be4199514d193"
Sep 11 18:20:07.237358 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.237296    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:07.240405 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.240390    2563 scope.go:117] "RemoveContainer" containerID="84a4c782e08061aa769412f37e1175c0d62ed280e37b37aedec77fcd4a28cca2"
Sep 11 18:20:07.243760 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.243740    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:07.246834 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.246819    2563 scope.go:117] "RemoveContainer" containerID="88470d7bacc41a1dfed0dcb2c14339ce55f69001d23af3bc2c4e8167723d4e49"
Sep 11 18:20:07.253546 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.253525    2563 scope.go:117] "RemoveContainer" containerID="1e93f0fe1f3f452ca1807acc8bf6d51579236b2ebfa709717e30ed8b7afb121c"
Sep 11 18:20:07.275422 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275399    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:07.275656 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275644    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy-thanos"
Sep 11 18:20:07.275699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275658    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy-thanos"
Sep 11 18:20:07.275699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275673    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="config-reloader"
Sep 11 18:20:07.275699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275678    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="config-reloader"
Sep 11 18:20:07.275699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275686    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy"
Sep 11 18:20:07.275699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275691    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275701    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="prometheus"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275706    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="prometheus"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275714    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy-web"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275720    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy-web"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275729    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="init-config-reloader"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275737    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="init-config-reloader"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275747    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="thanos-sidecar"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275755    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="thanos-sidecar"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275801    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275809    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="thanos-sidecar"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275815    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="config-reloader"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275820    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy-thanos"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275826    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="kube-rbac-proxy-web"
Sep 11 18:20:07.275845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.275831    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" containerName="prometheus"
Sep 11 18:20:07.280639 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.280625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.284124 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.284104    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 18:20:07.284218 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.284169    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 18:20:07.284283 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.284229    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 18:20:07.284544 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.284528    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 18:20:07.284712 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.284695    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 18:20:07.284807 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.284792    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n4qp4\""
Sep 11 18:20:07.284894 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.284880    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 18:20:07.285094 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.285075    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 18:20:07.285178 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.285165    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 18:20:07.285356 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.285341    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 18:20:07.285615 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.285564    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4c6u3d2jpudhh\""
Sep 11 18:20:07.285702 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.285627    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 18:20:07.285702 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.285643    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 18:20:07.286053 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.286019    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 18:20:07.290348 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.289396    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 18:20:07.293471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.293452    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 18:20:07.316390 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.302463    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:07.465865 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.465844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.465973 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.465874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e23f5a4-4c52-4091-91c4-127c8c834e08-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.465973 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.465895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.465973 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.465910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.465973 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.465924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466096 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.465980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466096 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466096 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466096 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-config\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466096 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466238 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466238 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466238 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466238 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466238 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466398 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466398 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e23f5a4-4c52-4091-91c4-127c8c834e08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466398 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466398 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.466398 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.466357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56wpv\" (UniqueName: \"kubernetes.io/projected/8e23f5a4-4c52-4091-91c4-127c8c834e08-kube-api-access-56wpv\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.566837 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.566813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.566933 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.566840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.566933 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.566861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.566933 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.566891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.566933 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.566919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567130 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.566949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e23f5a4-4c52-4091-91c4-127c8c834e08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567130 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.566975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567130 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.566998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567130 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56wpv\" (UniqueName: \"kubernetes.io/projected/8e23f5a4-4c52-4091-91c4-127c8c834e08-kube-api-access-56wpv\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567318 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567401 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567401 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e23f5a4-4c52-4091-91c4-127c8c834e08-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567569 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567569 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567569 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567710 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-config\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567710 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567710 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.567710 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.567676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.568933 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.568170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.568933 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.568354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.568933 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.568605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.568933 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.568783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.569442 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.569418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.569977 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.569952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.572255 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.570563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.572255 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.571257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.572255 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.571729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.572255 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.571783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.572255 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.571783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.572255 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.571812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e23f5a4-4c52-4091-91c4-127c8c834e08-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.572661 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.572325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e23f5a4-4c52-4091-91c4-127c8c834e08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.572723 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.572708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.573800 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.573773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-config\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.574005 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.573989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e23f5a4-4c52-4091-91c4-127c8c834e08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.574218 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.574199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e23f5a4-4c52-4091-91c4-127c8c834e08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.578795 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.578774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56wpv\" (UniqueName: \"kubernetes.io/projected/8e23f5a4-4c52-4091-91c4-127c8c834e08-kube-api-access-56wpv\") pod \"prometheus-k8s-0\" (UID: \"8e23f5a4-4c52-4091-91c4-127c8c834e08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.590517 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.590491    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:07.874108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:07.874079    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:07.874594 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:20:07.874573    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e23f5a4_4c52_4091_91c4_127c8c834e08.slice/crio-da27156ca8c3606315daf923061937564d3fea68a73d90d3e56247f7720d5a61 WatchSource:0}: Error finding container da27156ca8c3606315daf923061937564d3fea68a73d90d3e56247f7720d5a61: Status 404 returned error can't find the container with id da27156ca8c3606315daf923061937564d3fea68a73d90d3e56247f7720d5a61
Sep 11 18:20:08.218888 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:08.218854    2563 generic.go:356] "Generic (PLEG): container finished" podID="8e23f5a4-4c52-4091-91c4-127c8c834e08" containerID="ed654755e3660db89952880b997cca16ec2fea3a1827b9f4e23f84f4417088c9" exitCode=0
Sep 11 18:20:08.219197 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:08.218897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e23f5a4-4c52-4091-91c4-127c8c834e08","Type":"ContainerDied","Data":"ed654755e3660db89952880b997cca16ec2fea3a1827b9f4e23f84f4417088c9"}
Sep 11 18:20:08.219197 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:08.218962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e23f5a4-4c52-4091-91c4-127c8c834e08","Type":"ContainerStarted","Data":"da27156ca8c3606315daf923061937564d3fea68a73d90d3e56247f7720d5a61"}
Sep 11 18:20:09.112554 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:09.112530    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa5d02b-083f-445d-ac2b-cecca808874c" path="/var/lib/kubelet/pods/8aa5d02b-083f-445d-ac2b-cecca808874c/volumes"
Sep 11 18:20:09.225537 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:09.225515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e23f5a4-4c52-4091-91c4-127c8c834e08","Type":"ContainerStarted","Data":"cf830bdece6d38c38f2c4d6b22d40992c5571ccc905fb40aae4c88408ae4a9c7"}
Sep 11 18:20:09.225832 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:09.225563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e23f5a4-4c52-4091-91c4-127c8c834e08","Type":"ContainerStarted","Data":"5b94762d69c77bb6a6d7db29d8f41a7ffc56e0fd040379d0f8b682428eba58ef"}
Sep 11 18:20:09.225832 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:09.225592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e23f5a4-4c52-4091-91c4-127c8c834e08","Type":"ContainerStarted","Data":"0327bf10274b89c18d8e3c9abfc92c2daaab1b25b849bc1d8cef40157a675da7"}
Sep 11 18:20:09.225832 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:09.225616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e23f5a4-4c52-4091-91c4-127c8c834e08","Type":"ContainerStarted","Data":"2937b214f59d4c00e76443f0e9492fc8a7eb497656bc9074e11add42dda889b6"}
Sep 11 18:20:09.225832 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:09.225639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e23f5a4-4c52-4091-91c4-127c8c834e08","Type":"ContainerStarted","Data":"b3cdbf965789ac9d3270a411194eb4c28e7b24ae24de5ead6bf93b3a7f3eaf08"}
Sep 11 18:20:09.225832 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:09.225673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e23f5a4-4c52-4091-91c4-127c8c834e08","Type":"ContainerStarted","Data":"6540856a54548eeeeae31626a75c1a7065f13e9991c7a1fee83f8aea5c9dc6c1"}
Sep 11 18:20:09.262614 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:09.262571    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.239808267 podStartE2EDuration="2.262554222s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.745955ms" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:09.261121563 +0000 UTC m=+264.655636587" watchObservedRunningTime="2026-09-11 18:20:09.262554222 +0000 UTC m=+264.657069283"
Sep 11 18:20:12.590812 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:12.590777    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:45.021001 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:45.020971    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:20:45.024908 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:45.024885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:20:45.030118 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:45.030044    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:20:45.033344 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:45.033328    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 18:20:45.034214 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:20:45.034200    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:21:07.591583 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:21:07.591554    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:21:07.606090 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:21:07.606069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:21:08.457491 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:21:08.457469    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:22:24.445908 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.445877    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"]
Sep 11 18:22:24.448978 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.448957    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.453285 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.453258    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 18:22:24.453416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.453258    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:22:24.453416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.453316    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 18:22:24.453416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.453267    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 18:22:24.453416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.453259    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 18:22:24.453416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.453347    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tf62c\""
Sep 11 18:22:24.467521 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.467499    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"]
Sep 11 18:22:24.500307 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.500273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/77101585-1220-42ec-b540-98268146dc48-metrics-cert\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.500485 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.500343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/77101585-1220-42ec-b540-98268146dc48-manager-config\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.500567 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.500517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w879g\" (UniqueName: \"kubernetes.io/projected/77101585-1220-42ec-b540-98268146dc48-kube-api-access-w879g\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.500632 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.500569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77101585-1220-42ec-b540-98268146dc48-cert\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.601173 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.601149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/77101585-1220-42ec-b540-98268146dc48-metrics-cert\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.601305 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.601182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/77101585-1220-42ec-b540-98268146dc48-manager-config\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.601305 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.601214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w879g\" (UniqueName: \"kubernetes.io/projected/77101585-1220-42ec-b540-98268146dc48-kube-api-access-w879g\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.601305 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.601241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77101585-1220-42ec-b540-98268146dc48-cert\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.601782 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.601762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/77101585-1220-42ec-b540-98268146dc48-manager-config\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.603707 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.603686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/77101585-1220-42ec-b540-98268146dc48-metrics-cert\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.603781 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.603739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77101585-1220-42ec-b540-98268146dc48-cert\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.613588 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.613567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w879g\" (UniqueName: \"kubernetes.io/projected/77101585-1220-42ec-b540-98268146dc48-kube-api-access-w879g\") pod \"lws-controller-manager-68f8b56d64-9gd99\" (UID: \"77101585-1220-42ec-b540-98268146dc48\") " pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:24.762677 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:24.762622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:25.036491 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.036427    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"]
Sep 11 18:22:25.039166 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:22:25.039142    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77101585_1220_42ec_b540_98268146dc48.slice/crio-3ae1e979de93e4d4f7d0be1cba9dd93609459f0e9b7632a6d2f0cb5e31766105 WatchSource:0}: Error finding container 3ae1e979de93e4d4f7d0be1cba9dd93609459f0e9b7632a6d2f0cb5e31766105: Status 404 returned error can't find the container with id 3ae1e979de93e4d4f7d0be1cba9dd93609459f0e9b7632a6d2f0cb5e31766105
Sep 11 18:22:25.040933 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.040917    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:22:25.594414 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.594366    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"]
Sep 11 18:22:25.598500 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.598483    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.602180 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.602156    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 18:22:25.602618 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.602601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 18:22:25.602714 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.602638    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 18:22:25.603148 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.603132    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 18:22:25.603228 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.603199    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8779c\""
Sep 11 18:22:25.608413 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.608394    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c116a8f-4a34-4598-8347-f1d84eb51851-manifests\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.608507 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.608422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c116a8f-4a34-4598-8347-f1d84eb51851-webhook-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.608507 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.608449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c116a8f-4a34-4598-8347-f1d84eb51851-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.608507 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.608493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c116a8f-4a34-4598-8347-f1d84eb51851-tmp\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.608684 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.608513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79l8j\" (UniqueName: \"kubernetes.io/projected/4c116a8f-4a34-4598-8347-f1d84eb51851-kube-api-access-79l8j\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.615675 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.615656    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"]
Sep 11 18:22:25.709275 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.709254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c116a8f-4a34-4598-8347-f1d84eb51851-manifests\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.709400 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.709280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c116a8f-4a34-4598-8347-f1d84eb51851-webhook-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.709400 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.709303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c116a8f-4a34-4598-8347-f1d84eb51851-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.709400 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.709324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c116a8f-4a34-4598-8347-f1d84eb51851-tmp\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.709400 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.709345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79l8j\" (UniqueName: \"kubernetes.io/projected/4c116a8f-4a34-4598-8347-f1d84eb51851-kube-api-access-79l8j\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.709688 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.709665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c116a8f-4a34-4598-8347-f1d84eb51851-manifests\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.711536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.711515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c116a8f-4a34-4598-8347-f1d84eb51851-tmp\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.711821 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.711804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c116a8f-4a34-4598-8347-f1d84eb51851-webhook-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.711953 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.711932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c116a8f-4a34-4598-8347-f1d84eb51851-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.722653 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.722634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79l8j\" (UniqueName: \"kubernetes.io/projected/4c116a8f-4a34-4598-8347-f1d84eb51851-kube-api-access-79l8j\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-lx98r\" (UID: \"4c116a8f-4a34-4598-8347-f1d84eb51851\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:25.725615 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.725592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99" event={"ID":"77101585-1220-42ec-b540-98268146dc48","Type":"ContainerStarted","Data":"3ae1e979de93e4d4f7d0be1cba9dd93609459f0e9b7632a6d2f0cb5e31766105"}
Sep 11 18:22:25.914743 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:25.914715    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:26.219803 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:26.219764    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"]
Sep 11 18:22:26.223150 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:22:26.223120    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c116a8f_4a34_4598_8347_f1d84eb51851.slice/crio-c56a68858ed967c81fb83cc483f4be665225169ecc9615eac1e026e206f79973 WatchSource:0}: Error finding container c56a68858ed967c81fb83cc483f4be665225169ecc9615eac1e026e206f79973: Status 404 returned error can't find the container with id c56a68858ed967c81fb83cc483f4be665225169ecc9615eac1e026e206f79973
Sep 11 18:22:26.732195 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:26.732157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r" event={"ID":"4c116a8f-4a34-4598-8347-f1d84eb51851","Type":"ContainerStarted","Data":"c56a68858ed967c81fb83cc483f4be665225169ecc9615eac1e026e206f79973"}
Sep 11 18:22:27.739811 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:27.739718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99" event={"ID":"77101585-1220-42ec-b540-98268146dc48","Type":"ContainerStarted","Data":"3353ed596af2d6875d213478f13497c1880022bde436e88b386c710bda0c69db"}
Sep 11 18:22:27.740232 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:27.739958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:27.760673 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:27.760508    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99" podStartSLOduration=1.350825735 podStartE2EDuration="3.760487943s" totalImagesPullingTime="2.409662208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:27.759163551 +0000 UTC m=+403.153678604" watchObservedRunningTime="2026-09-11 18:22:27.760487943 +0000 UTC m=+403.155002967"
Sep 11 18:22:30.754980 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:30.754946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r" event={"ID":"4c116a8f-4a34-4598-8347-f1d84eb51851","Type":"ContainerStarted","Data":"db6718b9d8b053361a83419b2723bf3a9f6a41b18694ca98e900c7be52d14c55"}
Sep 11 18:22:31.760830 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:31.760770    2563 generic.go:356] "Generic (PLEG): container finished" podID="4c116a8f-4a34-4598-8347-f1d84eb51851" containerID="db6718b9d8b053361a83419b2723bf3a9f6a41b18694ca98e900c7be52d14c55" exitCode=0
Sep 11 18:22:31.760830 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:31.760817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r" event={"ID":"4c116a8f-4a34-4598-8347-f1d84eb51851","Type":"ContainerDied","Data":"db6718b9d8b053361a83419b2723bf3a9f6a41b18694ca98e900c7be52d14c55"}
Sep 11 18:22:32.767140 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:32.767108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r" event={"ID":"4c116a8f-4a34-4598-8347-f1d84eb51851","Type":"ContainerStarted","Data":"e51e7240a95ecd2f079590e69af758e3142712a0f5090f7544801d8e274d3d35"}
Sep 11 18:22:32.767503 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:32.767214    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:32.789483 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:32.789436    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r" podStartSLOduration=2.322444249 podStartE2EDuration="7.789418088s" totalImagesPullingTime="4.414327539s" totalInitContainerRuntime="1.0526463s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:32.787913542 +0000 UTC m=+408.182428593" watchObservedRunningTime="2026-09-11 18:22:32.789418088 +0000 UTC m=+408.183933115"
Sep 11 18:22:38.746591 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:38.746561    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68f8b56d64-9gd99"
Sep 11 18:22:43.773306 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:43.773273    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-lx98r"
Sep 11 18:22:45.813931 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:45.813898    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"]
Sep 11 18:22:45.886149 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:45.886119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"]
Sep 11 18:22:45.886322 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:45.886207    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"
Sep 11 18:22:45.888701 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:45.888682    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 18:22:45.888830 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:45.888809    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p25lq\""
Sep 11 18:22:45.889865 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:45.889850    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 18:22:45.955719 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:45.955695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbp7\" (UniqueName: \"kubernetes.io/projected/91e8d4a5-d3bc-4b60-83d3-43802ce56c06-kube-api-access-bjbp7\") pod \"kube-auth-proxy-78f85fb5d-hhxvv\" (UID: \"91e8d4a5-d3bc-4b60-83d3-43802ce56c06\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"
Sep 11 18:22:45.955824 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:45.955749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e8d4a5-d3bc-4b60-83d3-43802ce56c06-tls-certs\") pod \"kube-auth-proxy-78f85fb5d-hhxvv\" (UID: \"91e8d4a5-d3bc-4b60-83d3-43802ce56c06\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"
Sep 11 18:22:45.955824 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:45.955770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e8d4a5-d3bc-4b60-83d3-43802ce56c06-tmp\") pod \"kube-auth-proxy-78f85fb5d-hhxvv\" (UID: \"91e8d4a5-d3bc-4b60-83d3-43802ce56c06\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"
Sep 11 18:22:46.056668 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:46.056645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e8d4a5-d3bc-4b60-83d3-43802ce56c06-tls-certs\") pod \"kube-auth-proxy-78f85fb5d-hhxvv\" (UID: \"91e8d4a5-d3bc-4b60-83d3-43802ce56c06\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"
Sep 11 18:22:46.056788 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:46.056673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e8d4a5-d3bc-4b60-83d3-43802ce56c06-tmp\") pod \"kube-auth-proxy-78f85fb5d-hhxvv\" (UID: \"91e8d4a5-d3bc-4b60-83d3-43802ce56c06\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"
Sep 11 18:22:46.056788 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:46.056701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbp7\" (UniqueName: \"kubernetes.io/projected/91e8d4a5-d3bc-4b60-83d3-43802ce56c06-kube-api-access-bjbp7\") pod \"kube-auth-proxy-78f85fb5d-hhxvv\" (UID: \"91e8d4a5-d3bc-4b60-83d3-43802ce56c06\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"
Sep 11 18:22:46.058888 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:46.058866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e8d4a5-d3bc-4b60-83d3-43802ce56c06-tmp\") pod \"kube-auth-proxy-78f85fb5d-hhxvv\" (UID: \"91e8d4a5-d3bc-4b60-83d3-43802ce56c06\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"
Sep 11 18:22:46.059090 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:46.059071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e8d4a5-d3bc-4b60-83d3-43802ce56c06-tls-certs\") pod \"kube-auth-proxy-78f85fb5d-hhxvv\" (UID: \"91e8d4a5-d3bc-4b60-83d3-43802ce56c06\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"
Sep 11 18:22:46.065804 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:46.065752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbp7\" (UniqueName: \"kubernetes.io/projected/91e8d4a5-d3bc-4b60-83d3-43802ce56c06-kube-api-access-bjbp7\") pod \"kube-auth-proxy-78f85fb5d-hhxvv\" (UID: \"91e8d4a5-d3bc-4b60-83d3-43802ce56c06\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"
Sep 11 18:22:46.196339 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:46.196318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"
Sep 11 18:22:46.551914 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:46.551888    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv"]
Sep 11 18:22:46.554043 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:22:46.554017    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e8d4a5_d3bc_4b60_83d3_43802ce56c06.slice/crio-e2be80408eaa6378982719eddb03f5bc8d2ae0179780b66bb242eee9e0f18f6d WatchSource:0}: Error finding container e2be80408eaa6378982719eddb03f5bc8d2ae0179780b66bb242eee9e0f18f6d: Status 404 returned error can't find the container with id e2be80408eaa6378982719eddb03f5bc8d2ae0179780b66bb242eee9e0f18f6d
Sep 11 18:22:46.822224 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:46.822160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv" event={"ID":"91e8d4a5-d3bc-4b60-83d3-43802ce56c06","Type":"ContainerStarted","Data":"e2be80408eaa6378982719eddb03f5bc8d2ae0179780b66bb242eee9e0f18f6d"}
Sep 11 18:22:49.837719 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:49.837692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv" event={"ID":"91e8d4a5-d3bc-4b60-83d3-43802ce56c06","Type":"ContainerStarted","Data":"79203badd8868aed796a7c6a7138f1eaf6274c5a0c2b870c32a9c5ca5978571c"}
Sep 11 18:22:49.854974 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:22:49.854935    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78f85fb5d-hhxvv" podStartSLOduration=2.175364789 podStartE2EDuration="4.854919701s" totalImagesPullingTime="2.679554912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:49.853634145 +0000 UTC m=+425.248149191" watchObservedRunningTime="2026-09-11 18:22:49.854919701 +0000 UTC m=+425.249434726"
Sep 11 18:24:23.773444 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:23.773415    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"]
Sep 11 18:24:23.776715 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:23.776699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:23.784182 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:23.784144    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 18:24:23.784332 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:23.784297    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 18:24:23.785218 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:23.785195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8fzk7\""
Sep 11 18:24:23.790025 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:23.790006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"]
Sep 11 18:24:23.835208 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:23.835180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9hc\" (UniqueName: \"kubernetes.io/projected/881b4c55-c8fd-473e-99ed-68f7bcd8ec1d-kube-api-access-9l9hc\") pod \"limitador-operator-controller-manager-74495c69f8-m76gn\" (UID: \"881b4c55-c8fd-473e-99ed-68f7bcd8ec1d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:23.935522 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:23.935499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9hc\" (UniqueName: \"kubernetes.io/projected/881b4c55-c8fd-473e-99ed-68f7bcd8ec1d-kube-api-access-9l9hc\") pod \"limitador-operator-controller-manager-74495c69f8-m76gn\" (UID: \"881b4c55-c8fd-473e-99ed-68f7bcd8ec1d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:23.978344 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:23.978323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9hc\" (UniqueName: \"kubernetes.io/projected/881b4c55-c8fd-473e-99ed-68f7bcd8ec1d-kube-api-access-9l9hc\") pod \"limitador-operator-controller-manager-74495c69f8-m76gn\" (UID: \"881b4c55-c8fd-473e-99ed-68f7bcd8ec1d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:24.093130 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:24.093081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:24.362201 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:24.362181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"]
Sep 11 18:24:24.364389 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:24:24.364351    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881b4c55_c8fd_473e_99ed_68f7bcd8ec1d.slice/crio-dc9e10915b8ff8a0af4b777f07833aeb0059de4e094d5cdeacf83e4242d481ad WatchSource:0}: Error finding container dc9e10915b8ff8a0af4b777f07833aeb0059de4e094d5cdeacf83e4242d481ad: Status 404 returned error can't find the container with id dc9e10915b8ff8a0af4b777f07833aeb0059de4e094d5cdeacf83e4242d481ad
Sep 11 18:24:25.226149 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:25.226111    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn" event={"ID":"881b4c55-c8fd-473e-99ed-68f7bcd8ec1d","Type":"ContainerStarted","Data":"dc9e10915b8ff8a0af4b777f07833aeb0059de4e094d5cdeacf83e4242d481ad"}
Sep 11 18:24:27.236336 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:27.236305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn" event={"ID":"881b4c55-c8fd-473e-99ed-68f7bcd8ec1d","Type":"ContainerStarted","Data":"d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc"}
Sep 11 18:24:27.236781 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:27.236431    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:27.257831 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:27.257786    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn" podStartSLOduration=1.802321531 podStartE2EDuration="4.257772881s" totalImagesPullingTime="2.45545135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:27.256431523 +0000 UTC m=+522.650946571" watchObservedRunningTime="2026-09-11 18:24:27.257772881 +0000 UTC m=+522.652287898"
Sep 11 18:24:38.243416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:38.243327    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:44.700029 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.699997    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"]
Sep 11 18:24:44.700522 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.700224    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn" podUID="881b4c55-c8fd-473e-99ed-68f7bcd8ec1d" containerName="manager" containerID="cri-o://d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc" gracePeriod=2
Sep 11 18:24:44.708480 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.708453    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"]
Sep 11 18:24:44.759879 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.759852    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl"]
Sep 11 18:24:44.760249 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.760234    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881b4c55-c8fd-473e-99ed-68f7bcd8ec1d" containerName="manager"
Sep 11 18:24:44.760292 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.760251    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="881b4c55-c8fd-473e-99ed-68f7bcd8ec1d" containerName="manager"
Sep 11 18:24:44.760325 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.760316    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="881b4c55-c8fd-473e-99ed-68f7bcd8ec1d" containerName="manager"
Sep 11 18:24:44.763366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.763348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl"
Sep 11 18:24:44.766180 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:24:44.766155    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-m76gn\" is forbidden: User \"system:node:ip-10-0-143-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-92.ec2.internal' and this object" podUID="881b4c55-c8fd-473e-99ed-68f7bcd8ec1d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:44.792651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.792631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl"]
Sep 11 18:24:44.890898 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.890872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jpmp\" (UniqueName: \"kubernetes.io/projected/06aa86dd-e6e0-426b-93a7-d258487d21e1-kube-api-access-8jpmp\") pod \"limitador-operator-controller-manager-74495c69f8-2hjnl\" (UID: \"06aa86dd-e6e0-426b-93a7-d258487d21e1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl"
Sep 11 18:24:44.935349 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.935329    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:44.939110 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:24:44.939086    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-m76gn\" is forbidden: User \"system:node:ip-10-0-143-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-92.ec2.internal' and this object" podUID="881b4c55-c8fd-473e-99ed-68f7bcd8ec1d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:44.992172 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.992126    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l9hc\" (UniqueName: \"kubernetes.io/projected/881b4c55-c8fd-473e-99ed-68f7bcd8ec1d-kube-api-access-9l9hc\") pod \"881b4c55-c8fd-473e-99ed-68f7bcd8ec1d\" (UID: \"881b4c55-c8fd-473e-99ed-68f7bcd8ec1d\") "
Sep 11 18:24:44.992287 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.992273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jpmp\" (UniqueName: \"kubernetes.io/projected/06aa86dd-e6e0-426b-93a7-d258487d21e1-kube-api-access-8jpmp\") pod \"limitador-operator-controller-manager-74495c69f8-2hjnl\" (UID: \"06aa86dd-e6e0-426b-93a7-d258487d21e1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl"
Sep 11 18:24:44.994116 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:44.994092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881b4c55-c8fd-473e-99ed-68f7bcd8ec1d-kube-api-access-9l9hc" (OuterVolumeSpecName: "kube-api-access-9l9hc") pod "881b4c55-c8fd-473e-99ed-68f7bcd8ec1d" (UID: "881b4c55-c8fd-473e-99ed-68f7bcd8ec1d"). InnerVolumeSpecName "kube-api-access-9l9hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:24:45.017319 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:45.017298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jpmp\" (UniqueName: \"kubernetes.io/projected/06aa86dd-e6e0-426b-93a7-d258487d21e1-kube-api-access-8jpmp\") pod \"limitador-operator-controller-manager-74495c69f8-2hjnl\" (UID: \"06aa86dd-e6e0-426b-93a7-d258487d21e1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl"
Sep 11 18:24:45.085324 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:45.085303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl"
Sep 11 18:24:45.093202 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:45.093185    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l9hc\" (UniqueName: \"kubernetes.io/projected/881b4c55-c8fd-473e-99ed-68f7bcd8ec1d-kube-api-access-9l9hc\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:24:45.112141 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:24:45.112115    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-m76gn\" is forbidden: User \"system:node:ip-10-0-143-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-92.ec2.internal' and this object" podUID="881b4c55-c8fd-473e-99ed-68f7bcd8ec1d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:45.112862 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:45.112842    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="881b4c55-c8fd-473e-99ed-68f7bcd8ec1d" path="/var/lib/kubelet/pods/881b4c55-c8fd-473e-99ed-68f7bcd8ec1d/volumes"
Sep 11 18:24:45.312183 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:45.312106    2563 generic.go:356] "Generic (PLEG): container finished" podID="881b4c55-c8fd-473e-99ed-68f7bcd8ec1d" containerID="d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc" exitCode=0
Sep 11 18:24:45.312183 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:45.312154    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76gn"
Sep 11 18:24:45.312367 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:45.312203    2563 scope.go:117] "RemoveContainer" containerID="d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc"
Sep 11 18:24:45.319868 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:45.319842    2563 scope.go:117] "RemoveContainer" containerID="d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc"
Sep 11 18:24:45.320104 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:24:45.320086    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc\": container with ID starting with d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc not found: ID does not exist" containerID="d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc"
Sep 11 18:24:45.320152 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:45.320112    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc"} err="failed to get container status \"d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc\": rpc error: code = NotFound desc = could not find container \"d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc\": container with ID starting with d0b0ce5b8aa12f7c331a10671941055adce04ced8cfdc53fbc5c2564c25754fc not found: ID does not exist"
Sep 11 18:24:45.364676 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:45.364652    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl"]
Sep 11 18:24:45.368125 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:24:45.368102    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06aa86dd_e6e0_426b_93a7_d258487d21e1.slice/crio-8abbfc93d8ab757e2c8759ff347d2b928db5054ba80b266a272dc00662ef3db2 WatchSource:0}: Error finding container 8abbfc93d8ab757e2c8759ff347d2b928db5054ba80b266a272dc00662ef3db2: Status 404 returned error can't find the container with id 8abbfc93d8ab757e2c8759ff347d2b928db5054ba80b266a272dc00662ef3db2
Sep 11 18:24:46.319156 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:46.319126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl" event={"ID":"06aa86dd-e6e0-426b-93a7-d258487d21e1","Type":"ContainerStarted","Data":"f7c82c4ce69d93b3cd0e03979280505cf8488fee6c252b011ab8d9c6bd36b41a"}
Sep 11 18:24:46.319526 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:46.319171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl" event={"ID":"06aa86dd-e6e0-426b-93a7-d258487d21e1","Type":"ContainerStarted","Data":"8abbfc93d8ab757e2c8759ff347d2b928db5054ba80b266a272dc00662ef3db2"}
Sep 11 18:24:46.319526 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:46.319245    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl"
Sep 11 18:24:46.344445 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:46.344403    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl" podStartSLOduration=2.3443913370000002 podStartE2EDuration="2.344391337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:46.343102455 +0000 UTC m=+541.737617480" watchObservedRunningTime="2026-09-11 18:24:46.344391337 +0000 UTC m=+541.738906360"
Sep 11 18:24:57.325226 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:24:57.325195    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2hjnl"
Sep 11 18:25:28.652731 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.652695    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:25:28.656189 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.656167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x"
Sep 11 18:25:28.658765 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.658746    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 18:25:28.658765 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.658757    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hgmpm\""
Sep 11 18:25:28.665435 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.665413    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:25:28.694718 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.694698    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:25:28.798506 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.798473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5phkz\" (UniqueName: \"kubernetes.io/projected/3495b1cb-0e6d-4183-b202-9b307262e441-kube-api-access-5phkz\") pod \"limitador-limitador-84994f7ffc-xv98x\" (UID: \"3495b1cb-0e6d-4183-b202-9b307262e441\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x"
Sep 11 18:25:28.798597 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.798523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3495b1cb-0e6d-4183-b202-9b307262e441-config-file\") pod \"limitador-limitador-84994f7ffc-xv98x\" (UID: \"3495b1cb-0e6d-4183-b202-9b307262e441\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x"
Sep 11 18:25:28.899314 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.899287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5phkz\" (UniqueName: \"kubernetes.io/projected/3495b1cb-0e6d-4183-b202-9b307262e441-kube-api-access-5phkz\") pod \"limitador-limitador-84994f7ffc-xv98x\" (UID: \"3495b1cb-0e6d-4183-b202-9b307262e441\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x"
Sep 11 18:25:28.899436 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.899350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3495b1cb-0e6d-4183-b202-9b307262e441-config-file\") pod \"limitador-limitador-84994f7ffc-xv98x\" (UID: \"3495b1cb-0e6d-4183-b202-9b307262e441\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x"
Sep 11 18:25:28.899929 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.899894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3495b1cb-0e6d-4183-b202-9b307262e441-config-file\") pod \"limitador-limitador-84994f7ffc-xv98x\" (UID: \"3495b1cb-0e6d-4183-b202-9b307262e441\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x"
Sep 11 18:25:28.907432 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.907412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5phkz\" (UniqueName: \"kubernetes.io/projected/3495b1cb-0e6d-4183-b202-9b307262e441-kube-api-access-5phkz\") pod \"limitador-limitador-84994f7ffc-xv98x\" (UID: \"3495b1cb-0e6d-4183-b202-9b307262e441\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x"
Sep 11 18:25:28.966387 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:28.966349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x"
Sep 11 18:25:29.196032 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:29.195960    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-5g95g"]
Sep 11 18:25:29.199314 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:29.199299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5g95g"
Sep 11 18:25:29.202156 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:29.202136    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2wp7p\""
Sep 11 18:25:29.206124 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:29.206100    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5g95g"]
Sep 11 18:25:29.236358 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:29.236340    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:25:29.239058 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:25:29.239036    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3495b1cb_0e6d_4183_b202_9b307262e441.slice/crio-3a7f87ac266d879ee1d3d5b9673c0d115c0d77756131b04bb02126c84203188f WatchSource:0}: Error finding container 3a7f87ac266d879ee1d3d5b9673c0d115c0d77756131b04bb02126c84203188f: Status 404 returned error can't find the container with id 3a7f87ac266d879ee1d3d5b9673c0d115c0d77756131b04bb02126c84203188f
Sep 11 18:25:29.302877 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:29.302855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p862\" (UniqueName: \"kubernetes.io/projected/d9cf776a-dc91-492c-b855-4a185353b57f-kube-api-access-8p862\") pod \"authorino-59c974bd48-5g95g\" (UID: \"d9cf776a-dc91-492c-b855-4a185353b57f\") " pod="kuadrant-system/authorino-59c974bd48-5g95g"
Sep 11 18:25:29.403180 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:29.403158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p862\" (UniqueName: \"kubernetes.io/projected/d9cf776a-dc91-492c-b855-4a185353b57f-kube-api-access-8p862\") pod \"authorino-59c974bd48-5g95g\" (UID: \"d9cf776a-dc91-492c-b855-4a185353b57f\") " pod="kuadrant-system/authorino-59c974bd48-5g95g"
Sep 11 18:25:29.413052 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:29.413031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p862\" (UniqueName: \"kubernetes.io/projected/d9cf776a-dc91-492c-b855-4a185353b57f-kube-api-access-8p862\") pod \"authorino-59c974bd48-5g95g\" (UID: \"d9cf776a-dc91-492c-b855-4a185353b57f\") " pod="kuadrant-system/authorino-59c974bd48-5g95g"
Sep 11 18:25:29.500871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:29.500816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x" event={"ID":"3495b1cb-0e6d-4183-b202-9b307262e441","Type":"ContainerStarted","Data":"3a7f87ac266d879ee1d3d5b9673c0d115c0d77756131b04bb02126c84203188f"}
Sep 11 18:25:29.517092 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:29.517064    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5g95g"
Sep 11 18:25:29.782911 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:29.782885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5g95g"]
Sep 11 18:25:29.785190 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:25:29.785166    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9cf776a_dc91_492c_b855_4a185353b57f.slice/crio-e2da5da08f8129a833fa7a8e49d76c17cdd4610cea8c5c6ea75bf74fb15dafc4 WatchSource:0}: Error finding container e2da5da08f8129a833fa7a8e49d76c17cdd4610cea8c5c6ea75bf74fb15dafc4: Status 404 returned error can't find the container with id e2da5da08f8129a833fa7a8e49d76c17cdd4610cea8c5c6ea75bf74fb15dafc4
Sep 11 18:25:30.512801 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:30.508112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5g95g" event={"ID":"d9cf776a-dc91-492c-b855-4a185353b57f","Type":"ContainerStarted","Data":"e2da5da08f8129a833fa7a8e49d76c17cdd4610cea8c5c6ea75bf74fb15dafc4"}
Sep 11 18:25:32.519307 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:32.519268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x" event={"ID":"3495b1cb-0e6d-4183-b202-9b307262e441","Type":"ContainerStarted","Data":"b2e8592cfafdd308119d88ba1c0429a1c433dfc70b8b1523cc1a25fd83e0bb80"}
Sep 11 18:25:32.519751 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:32.519338    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x"
Sep 11 18:25:32.520950 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:32.520925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5g95g" event={"ID":"d9cf776a-dc91-492c-b855-4a185353b57f","Type":"ContainerStarted","Data":"63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02"}
Sep 11 18:25:32.537477 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:32.537430    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x" podStartSLOduration=2.321319045 podStartE2EDuration="4.537416005s" totalImagesPullingTime="2.21609696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:32.536532303 +0000 UTC m=+587.931047327" watchObservedRunningTime="2026-09-11 18:25:32.537416005 +0000 UTC m=+587.931931028"
Sep 11 18:25:32.550948 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:32.550913    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-5g95g" podStartSLOduration=1.87652845 podStartE2EDuration="3.55090166s" totalImagesPullingTime="1.67437321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:32.550661215 +0000 UTC m=+587.945176240" watchObservedRunningTime="2026-09-11 18:25:32.55090166 +0000 UTC m=+587.945416685"
Sep 11 18:25:43.526896 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:43.526866    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-xv98x"
Sep 11 18:25:45.053280 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:45.053253    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:25:45.053698 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:45.053586    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:25:45.062039 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:45.062013    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:25:45.062168 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:45.062069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:25:58.555776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:58.555742    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-l7dxp"]
Sep 11 18:25:58.560137 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:58.560116    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-l7dxp"
Sep 11 18:25:58.567916 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:58.567893    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-l7dxp"]
Sep 11 18:25:58.706272 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:58.706247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgsl4\" (UniqueName: \"kubernetes.io/projected/7035eed8-2bc9-420a-bb91-46cd76768d05-kube-api-access-jgsl4\") pod \"authorino-557f876f69-l7dxp\" (UID: \"7035eed8-2bc9-420a-bb91-46cd76768d05\") " pod="kuadrant-system/authorino-557f876f69-l7dxp"
Sep 11 18:25:58.783327 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:58.783308    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-l7dxp"]
Sep 11 18:25:58.783507 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:25:58.783489    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jgsl4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-l7dxp" podUID="7035eed8-2bc9-420a-bb91-46cd76768d05"
Sep 11 18:25:58.805061 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:58.805040    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bf69974c8-kh42g"]
Sep 11 18:25:58.806965 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:58.806917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgsl4\" (UniqueName: \"kubernetes.io/projected/7035eed8-2bc9-420a-bb91-46cd76768d05-kube-api-access-jgsl4\") pod \"authorino-557f876f69-l7dxp\" (UID: \"7035eed8-2bc9-420a-bb91-46cd76768d05\") " pod="kuadrant-system/authorino-557f876f69-l7dxp"
Sep 11 18:25:58.808221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:58.808206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bf69974c8-kh42g"
Sep 11 18:25:58.816753 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:58.816730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bf69974c8-kh42g"]
Sep 11 18:25:58.818680 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:58.818661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgsl4\" (UniqueName: \"kubernetes.io/projected/7035eed8-2bc9-420a-bb91-46cd76768d05-kube-api-access-jgsl4\") pod \"authorino-557f876f69-l7dxp\" (UID: \"7035eed8-2bc9-420a-bb91-46cd76768d05\") " pod="kuadrant-system/authorino-557f876f69-l7dxp"
Sep 11 18:25:58.907884 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:58.907863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9wm7\" (UniqueName: \"kubernetes.io/projected/5826b42b-6523-45ec-820f-a52b0a8d87aa-kube-api-access-n9wm7\") pod \"authorino-bf69974c8-kh42g\" (UID: \"5826b42b-6523-45ec-820f-a52b0a8d87aa\") " pod="kuadrant-system/authorino-bf69974c8-kh42g"
Sep 11 18:25:59.008683 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:59.008652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9wm7\" (UniqueName: \"kubernetes.io/projected/5826b42b-6523-45ec-820f-a52b0a8d87aa-kube-api-access-n9wm7\") pod \"authorino-bf69974c8-kh42g\" (UID: \"5826b42b-6523-45ec-820f-a52b0a8d87aa\") " pod="kuadrant-system/authorino-bf69974c8-kh42g"
Sep 11 18:25:59.016180 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:59.016164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9wm7\" (UniqueName: \"kubernetes.io/projected/5826b42b-6523-45ec-820f-a52b0a8d87aa-kube-api-access-n9wm7\") pod \"authorino-bf69974c8-kh42g\" (UID: \"5826b42b-6523-45ec-820f-a52b0a8d87aa\") " pod="kuadrant-system/authorino-bf69974c8-kh42g"
Sep 11 18:25:59.117258 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:59.117177    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bf69974c8-kh42g"
Sep 11 18:25:59.384907 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:59.384880    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bf69974c8-kh42g"]
Sep 11 18:25:59.387464 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:25:59.387436    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5826b42b_6523_45ec_820f_a52b0a8d87aa.slice/crio-3fe655c3953ef7717a5cb81b8d884942745ab0c6d021eea02bf759b78a0ac10a WatchSource:0}: Error finding container 3fe655c3953ef7717a5cb81b8d884942745ab0c6d021eea02bf759b78a0ac10a: Status 404 returned error can't find the container with id 3fe655c3953ef7717a5cb81b8d884942745ab0c6d021eea02bf759b78a0ac10a
Sep 11 18:25:59.642475 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:59.642390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bf69974c8-kh42g" event={"ID":"5826b42b-6523-45ec-820f-a52b0a8d87aa","Type":"ContainerStarted","Data":"3fe655c3953ef7717a5cb81b8d884942745ab0c6d021eea02bf759b78a0ac10a"}
Sep 11 18:25:59.642475 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:59.642422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-l7dxp"
Sep 11 18:25:59.650477 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:59.650456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-l7dxp"
Sep 11 18:25:59.719693 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:59.719674    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgsl4\" (UniqueName: \"kubernetes.io/projected/7035eed8-2bc9-420a-bb91-46cd76768d05-kube-api-access-jgsl4\") pod \"7035eed8-2bc9-420a-bb91-46cd76768d05\" (UID: \"7035eed8-2bc9-420a-bb91-46cd76768d05\") "
Sep 11 18:25:59.721589 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:59.721565    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7035eed8-2bc9-420a-bb91-46cd76768d05-kube-api-access-jgsl4" (OuterVolumeSpecName: "kube-api-access-jgsl4") pod "7035eed8-2bc9-420a-bb91-46cd76768d05" (UID: "7035eed8-2bc9-420a-bb91-46cd76768d05"). InnerVolumeSpecName "kube-api-access-jgsl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:25:59.820479 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:25:59.820452    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgsl4\" (UniqueName: \"kubernetes.io/projected/7035eed8-2bc9-420a-bb91-46cd76768d05-kube-api-access-jgsl4\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:26:00.648155 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.648128    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-l7dxp"
Sep 11 18:26:00.648557 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.648124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bf69974c8-kh42g" event={"ID":"5826b42b-6523-45ec-820f-a52b0a8d87aa","Type":"ContainerStarted","Data":"6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298"}
Sep 11 18:26:00.665431 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.665392    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bf69974c8-kh42g" podStartSLOduration=2.121679173 podStartE2EDuration="2.665363507s" totalImagesPullingTime="543.684334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:00.664316146 +0000 UTC m=+616.058831171" watchObservedRunningTime="2026-09-11 18:26:00.665363507 +0000 UTC m=+616.059878526"
Sep 11 18:26:00.688417 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.688391    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-l7dxp"]
Sep 11 18:26:00.695417 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.695390    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-l7dxp"]
Sep 11 18:26:00.696678 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.696655    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5g95g"]
Sep 11 18:26:00.696839 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.696818    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-5g95g" podUID="d9cf776a-dc91-492c-b855-4a185353b57f" containerName="authorino" containerID="cri-o://63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02" gracePeriod=30
Sep 11 18:26:00.773357 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.773330    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dd9fcb464-4thh8"]
Sep 11 18:26:00.776409 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.776395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dd9fcb464-4thh8"
Sep 11 18:26:00.778842 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.778824    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 18:26:00.783221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.783052    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dd9fcb464-4thh8"]
Sep 11 18:26:00.829027 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.829002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbcbm\" (UniqueName: \"kubernetes.io/projected/81951027-c972-4740-a697-55a6faae04bc-kube-api-access-kbcbm\") pod \"authorino-5dd9fcb464-4thh8\" (UID: \"81951027-c972-4740-a697-55a6faae04bc\") " pod="kuadrant-system/authorino-5dd9fcb464-4thh8"
Sep 11 18:26:00.829143 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.829096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81951027-c972-4740-a697-55a6faae04bc-tls-cert\") pod \"authorino-5dd9fcb464-4thh8\" (UID: \"81951027-c972-4740-a697-55a6faae04bc\") " pod="kuadrant-system/authorino-5dd9fcb464-4thh8"
Sep 11 18:26:00.930138 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.930114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbcbm\" (UniqueName: \"kubernetes.io/projected/81951027-c972-4740-a697-55a6faae04bc-kube-api-access-kbcbm\") pod \"authorino-5dd9fcb464-4thh8\" (UID: \"81951027-c972-4740-a697-55a6faae04bc\") " pod="kuadrant-system/authorino-5dd9fcb464-4thh8"
Sep 11 18:26:00.930273 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.930255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81951027-c972-4740-a697-55a6faae04bc-tls-cert\") pod \"authorino-5dd9fcb464-4thh8\" (UID: \"81951027-c972-4740-a697-55a6faae04bc\") " pod="kuadrant-system/authorino-5dd9fcb464-4thh8"
Sep 11 18:26:00.932769 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.932747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81951027-c972-4740-a697-55a6faae04bc-tls-cert\") pod \"authorino-5dd9fcb464-4thh8\" (UID: \"81951027-c972-4740-a697-55a6faae04bc\") " pod="kuadrant-system/authorino-5dd9fcb464-4thh8"
Sep 11 18:26:00.937851 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.937829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbcbm\" (UniqueName: \"kubernetes.io/projected/81951027-c972-4740-a697-55a6faae04bc-kube-api-access-kbcbm\") pod \"authorino-5dd9fcb464-4thh8\" (UID: \"81951027-c972-4740-a697-55a6faae04bc\") " pod="kuadrant-system/authorino-5dd9fcb464-4thh8"
Sep 11 18:26:00.951839 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:00.951819    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5g95g"
Sep 11 18:26:01.086698 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.086671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dd9fcb464-4thh8"
Sep 11 18:26:01.113807 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.113781    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7035eed8-2bc9-420a-bb91-46cd76768d05" path="/var/lib/kubelet/pods/7035eed8-2bc9-420a-bb91-46cd76768d05/volumes"
Sep 11 18:26:01.132183 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.132155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p862\" (UniqueName: \"kubernetes.io/projected/d9cf776a-dc91-492c-b855-4a185353b57f-kube-api-access-8p862\") pod \"d9cf776a-dc91-492c-b855-4a185353b57f\" (UID: \"d9cf776a-dc91-492c-b855-4a185353b57f\") "
Sep 11 18:26:01.134944 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.134887    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9cf776a-dc91-492c-b855-4a185353b57f-kube-api-access-8p862" (OuterVolumeSpecName: "kube-api-access-8p862") pod "d9cf776a-dc91-492c-b855-4a185353b57f" (UID: "d9cf776a-dc91-492c-b855-4a185353b57f"). InnerVolumeSpecName "kube-api-access-8p862". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:26:01.233760 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.233699    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p862\" (UniqueName: \"kubernetes.io/projected/d9cf776a-dc91-492c-b855-4a185353b57f-kube-api-access-8p862\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:26:01.356480 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.356363    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dd9fcb464-4thh8"]
Sep 11 18:26:01.358954 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:26:01.358926    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81951027_c972_4740_a697_55a6faae04bc.slice/crio-34d461ddcb0899e94daaef0b21d7917b34dbb147cc4a793cb6aa2e8740dd15d4 WatchSource:0}: Error finding container 34d461ddcb0899e94daaef0b21d7917b34dbb147cc4a793cb6aa2e8740dd15d4: Status 404 returned error can't find the container with id 34d461ddcb0899e94daaef0b21d7917b34dbb147cc4a793cb6aa2e8740dd15d4
Sep 11 18:26:01.653459 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.653429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dd9fcb464-4thh8" event={"ID":"81951027-c972-4740-a697-55a6faae04bc","Type":"ContainerStarted","Data":"34d461ddcb0899e94daaef0b21d7917b34dbb147cc4a793cb6aa2e8740dd15d4"}
Sep 11 18:26:01.654722 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.654701    2563 generic.go:356] "Generic (PLEG): container finished" podID="d9cf776a-dc91-492c-b855-4a185353b57f" containerID="63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02" exitCode=0
Sep 11 18:26:01.654808 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.654741    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5g95g"
Sep 11 18:26:01.654808 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.654780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5g95g" event={"ID":"d9cf776a-dc91-492c-b855-4a185353b57f","Type":"ContainerDied","Data":"63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02"}
Sep 11 18:26:01.654873 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.654817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5g95g" event={"ID":"d9cf776a-dc91-492c-b855-4a185353b57f","Type":"ContainerDied","Data":"e2da5da08f8129a833fa7a8e49d76c17cdd4610cea8c5c6ea75bf74fb15dafc4"}
Sep 11 18:26:01.654873 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.654841    2563 scope.go:117] "RemoveContainer" containerID="63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02"
Sep 11 18:26:01.663077 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.663055    2563 scope.go:117] "RemoveContainer" containerID="63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02"
Sep 11 18:26:01.663319 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:26:01.663301    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02\": container with ID starting with 63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02 not found: ID does not exist" containerID="63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02"
Sep 11 18:26:01.663403 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.663325    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02"} err="failed to get container status \"63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02\": rpc error: code = NotFound desc = could not find container \"63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02\": container with ID starting with 63ad751c3613d3648b9492918f5d387030da32e88c5d8c104b9570d05acb7c02 not found: ID does not exist"
Sep 11 18:26:01.675100 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.675079    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5g95g"]
Sep 11 18:26:01.678914 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:01.678886    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5g95g"]
Sep 11 18:26:02.660452 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:02.660398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dd9fcb464-4thh8" event={"ID":"81951027-c972-4740-a697-55a6faae04bc","Type":"ContainerStarted","Data":"082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c"}
Sep 11 18:26:02.676095 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:02.676051    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dd9fcb464-4thh8" podStartSLOduration=1.816514378 podStartE2EDuration="2.676035806s" totalImagesPullingTime="859.521428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:02.675685849 +0000 UTC m=+618.070200873" watchObservedRunningTime="2026-09-11 18:26:02.676035806 +0000 UTC m=+618.070550829"
Sep 11 18:26:02.699767 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:02.699742    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bf69974c8-kh42g"]
Sep 11 18:26:02.699884 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:02.699867    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-bf69974c8-kh42g" podUID="5826b42b-6523-45ec-820f-a52b0a8d87aa" containerName="authorino" containerID="cri-o://6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298" gracePeriod=30
Sep 11 18:26:02.947064 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:02.947042    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bf69974c8-kh42g"
Sep 11 18:26:03.044073 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.044049    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9wm7\" (UniqueName: \"kubernetes.io/projected/5826b42b-6523-45ec-820f-a52b0a8d87aa-kube-api-access-n9wm7\") pod \"5826b42b-6523-45ec-820f-a52b0a8d87aa\" (UID: \"5826b42b-6523-45ec-820f-a52b0a8d87aa\") "
Sep 11 18:26:03.045941 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.045919    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5826b42b-6523-45ec-820f-a52b0a8d87aa-kube-api-access-n9wm7" (OuterVolumeSpecName: "kube-api-access-n9wm7") pod "5826b42b-6523-45ec-820f-a52b0a8d87aa" (UID: "5826b42b-6523-45ec-820f-a52b0a8d87aa"). InnerVolumeSpecName "kube-api-access-n9wm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:26:03.112014 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.111988    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9cf776a-dc91-492c-b855-4a185353b57f" path="/var/lib/kubelet/pods/d9cf776a-dc91-492c-b855-4a185353b57f/volumes"
Sep 11 18:26:03.145075 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.145056    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9wm7\" (UniqueName: \"kubernetes.io/projected/5826b42b-6523-45ec-820f-a52b0a8d87aa-kube-api-access-n9wm7\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:26:03.666701 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.666671    2563 generic.go:356] "Generic (PLEG): container finished" podID="5826b42b-6523-45ec-820f-a52b0a8d87aa" containerID="6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298" exitCode=0
Sep 11 18:26:03.667138 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.666716    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bf69974c8-kh42g"
Sep 11 18:26:03.667138 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.666754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bf69974c8-kh42g" event={"ID":"5826b42b-6523-45ec-820f-a52b0a8d87aa","Type":"ContainerDied","Data":"6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298"}
Sep 11 18:26:03.667138 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.666793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bf69974c8-kh42g" event={"ID":"5826b42b-6523-45ec-820f-a52b0a8d87aa","Type":"ContainerDied","Data":"3fe655c3953ef7717a5cb81b8d884942745ab0c6d021eea02bf759b78a0ac10a"}
Sep 11 18:26:03.667138 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.666810    2563 scope.go:117] "RemoveContainer" containerID="6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298"
Sep 11 18:26:03.674710 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.674693    2563 scope.go:117] "RemoveContainer" containerID="6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298"
Sep 11 18:26:03.674947 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:26:03.674930    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298\": container with ID starting with 6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298 not found: ID does not exist" containerID="6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298"
Sep 11 18:26:03.674989 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.674955    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298"} err="failed to get container status \"6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298\": rpc error: code = NotFound desc = could not find container \"6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298\": container with ID starting with 6e5f5c7b0d9f27f3f2391144b39fdbf5704fda3d5a19eedf912fc2aee3bec298 not found: ID does not exist"
Sep 11 18:26:03.682104 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.682072    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bf69974c8-kh42g"]
Sep 11 18:26:03.687095 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:03.687074    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bf69974c8-kh42g"]
Sep 11 18:26:05.111863 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:05.111836    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5826b42b-6523-45ec-820f-a52b0a8d87aa" path="/var/lib/kubelet/pods/5826b42b-6523-45ec-820f-a52b0a8d87aa/volumes"
Sep 11 18:26:45.084012 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:26:45.083985    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:27:10.638723 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:27:10.638678    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:27:36.947204 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:27:36.947164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:27:46.244557 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:27:46.244515    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:27:55.542875 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:27:55.542841    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:28:00.242458 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.242401    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"]
Sep 11 18:28:00.242896 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.242878    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9cf776a-dc91-492c-b855-4a185353b57f" containerName="authorino"
Sep 11 18:28:00.242962 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.242897    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9cf776a-dc91-492c-b855-4a185353b57f" containerName="authorino"
Sep 11 18:28:00.242962 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.242920    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5826b42b-6523-45ec-820f-a52b0a8d87aa" containerName="authorino"
Sep 11 18:28:00.242962 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.242928    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5826b42b-6523-45ec-820f-a52b0a8d87aa" containerName="authorino"
Sep 11 18:28:00.243101 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.243029    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5826b42b-6523-45ec-820f-a52b0a8d87aa" containerName="authorino"
Sep 11 18:28:00.243101 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.243049    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9cf776a-dc91-492c-b855-4a185353b57f" containerName="authorino"
Sep 11 18:28:00.246120 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.246100    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.248569 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.248548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 18:28:00.249699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.249674    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6klqx\""
Sep 11 18:28:00.249811 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.249709    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 18:28:00.249811 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.249756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 18:28:00.255784 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.255760    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"]
Sep 11 18:28:00.270572 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.270542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.270665 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.270592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e628f0d-70d0-4397-b6c4-298b4dab91e5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.270665 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.270641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.270745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.270699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.270745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.270726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.270824 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.270762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx7wj\" (UniqueName: \"kubernetes.io/projected/2e628f0d-70d0-4397-b6c4-298b4dab91e5-kube-api-access-kx7wj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.371322 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.371291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e628f0d-70d0-4397-b6c4-298b4dab91e5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.371501 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.371332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.371501 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.371354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.371501 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.371403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.371501 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.371426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx7wj\" (UniqueName: \"kubernetes.io/projected/2e628f0d-70d0-4397-b6c4-298b4dab91e5-kube-api-access-kx7wj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.371501 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.371478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.371799 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.371773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.371846 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.371807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.371897 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.371875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.373608 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.373583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e628f0d-70d0-4397-b6c4-298b4dab91e5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.373851 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.373835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e628f0d-70d0-4397-b6c4-298b4dab91e5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.379449 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.379425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx7wj\" (UniqueName: \"kubernetes.io/projected/2e628f0d-70d0-4397-b6c4-298b4dab91e5-kube-api-access-kx7wj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp\" (UID: \"2e628f0d-70d0-4397-b6c4-298b4dab91e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.557122 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.557060    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:00.827346 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.827319    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"]
Sep 11 18:28:00.829211 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:28:00.829184    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e628f0d_70d0_4397_b6c4_298b4dab91e5.slice/crio-97f647947cb1890ce0ce9aba4ebbf5feaaf820cd5a82d5467330e47572a73d2a WatchSource:0}: Error finding container 97f647947cb1890ce0ce9aba4ebbf5feaaf820cd5a82d5467330e47572a73d2a: Status 404 returned error can't find the container with id 97f647947cb1890ce0ce9aba4ebbf5feaaf820cd5a82d5467330e47572a73d2a
Sep 11 18:28:00.830628 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:00.830604    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:28:01.169383 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:01.169321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp" event={"ID":"2e628f0d-70d0-4397-b6c4-298b4dab91e5","Type":"ContainerStarted","Data":"97f647947cb1890ce0ce9aba4ebbf5feaaf820cd5a82d5467330e47572a73d2a"}
Sep 11 18:28:01.849163 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:01.849135    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:28:02.176144 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:02.176109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp" event={"ID":"2e628f0d-70d0-4397-b6c4-298b4dab91e5","Type":"ContainerStarted","Data":"b8ea30c0784ca674f5c35bb09ba97a449d5bf3816a9cbd2afc56982939b8a959"}
Sep 11 18:28:02.197574 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:02.197520    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp" podStartSLOduration=1.204030964 podStartE2EDuration="2.197504401s" totalImagesPullingTime="993.473437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:02.196859972 +0000 UTC m=+737.591375018" watchObservedRunningTime="2026-09-11 18:28:02.197504401 +0000 UTC m=+737.592019424"
Sep 11 18:28:10.557930 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:10.557897    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:10.558299 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:10.557957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:10.570403 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:10.570355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:11.223718 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:11.223691    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp"
Sep 11 18:28:22.744280 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.744247    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"]
Sep 11 18:28:22.747771 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.747749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.750195 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.750170    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 18:28:22.759188 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.759168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"]
Sep 11 18:28:22.844162 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.844137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.844279 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.844178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzh72\" (UniqueName: \"kubernetes.io/projected/0f92b956-3605-4b83-b672-8c9f5c774f9f-kube-api-access-dzh72\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.844279 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.844198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.844279 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.844252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f92b956-3605-4b83-b672-8c9f5c774f9f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.844455 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.844279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.844455 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.844308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.944957 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.944931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.945058 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.944970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzh72\" (UniqueName: \"kubernetes.io/projected/0f92b956-3605-4b83-b672-8c9f5c774f9f-kube-api-access-dzh72\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.945058 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.944988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.945058 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.945019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f92b956-3605-4b83-b672-8c9f5c774f9f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.945058 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.945044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.945199 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.945080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.945359 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.945337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.945474 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.945452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.945520 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.945457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.947368 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.947343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f92b956-3605-4b83-b672-8c9f5c774f9f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.947555 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.947536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f92b956-3605-4b83-b672-8c9f5c774f9f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:22.953090 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:22.953073    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzh72\" (UniqueName: \"kubernetes.io/projected/0f92b956-3605-4b83-b672-8c9f5c774f9f-kube-api-access-dzh72\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jsshp\" (UID: \"0f92b956-3605-4b83-b672-8c9f5c774f9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:23.057894 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:23.057835    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:23.329628 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:23.329603    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"]
Sep 11 18:28:23.331290 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:28:23.331262    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f92b956_3605_4b83_b672_8c9f5c774f9f.slice/crio-a7e0a1b50825f03f258c1e8a45d8ec2dad108f335ff03841ef2e55ffc7089270 WatchSource:0}: Error finding container a7e0a1b50825f03f258c1e8a45d8ec2dad108f335ff03841ef2e55ffc7089270: Status 404 returned error can't find the container with id a7e0a1b50825f03f258c1e8a45d8ec2dad108f335ff03841ef2e55ffc7089270
Sep 11 18:28:24.271017 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:24.270972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp" event={"ID":"0f92b956-3605-4b83-b672-8c9f5c774f9f","Type":"ContainerStarted","Data":"19fd5048b68a7a40803289d5f01e72759e443a7263cdc0210a07b1b88eb2480c"}
Sep 11 18:28:24.271352 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:24.271046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp" event={"ID":"0f92b956-3605-4b83-b672-8c9f5c774f9f","Type":"ContainerStarted","Data":"a7e0a1b50825f03f258c1e8a45d8ec2dad108f335ff03841ef2e55ffc7089270"}
Sep 11 18:28:24.288181 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:24.288139    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp" podStartSLOduration=2.102423082 podStartE2EDuration="2.288126051s" totalImagesPullingTime="185.702969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:24.286769477 +0000 UTC m=+759.681284512" watchObservedRunningTime="2026-09-11 18:28:24.288126051 +0000 UTC m=+759.682641066"
Sep 11 18:28:24.544988 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:24.544918    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:28:27.939042 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:27.939008    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"]
Sep 11 18:28:27.942297 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:27.942273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:27.944768 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:27.944746    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 18:28:27.952727 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:27.952707    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"]
Sep 11 18:28:28.083073 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.083049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.083196 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.083099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ed16e3-328f-4143-9d0e-200121d00c4d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.083196 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.083120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg9tv\" (UniqueName: \"kubernetes.io/projected/b5ed16e3-328f-4143-9d0e-200121d00c4d-kube-api-access-lg9tv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.083307 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.083199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.083307 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.083260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.083307 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.083297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.184330 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.184292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.184472 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.184341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.184472 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.184400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ed16e3-328f-4143-9d0e-200121d00c4d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.184564 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.184518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9tv\" (UniqueName: \"kubernetes.io/projected/b5ed16e3-328f-4143-9d0e-200121d00c4d-kube-api-access-lg9tv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.184616 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.184581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.184616 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.184597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.184710 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.184629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.184710 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.184671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.184921 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.184903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.186609 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.186590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ed16e3-328f-4143-9d0e-200121d00c4d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.186845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.186830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ed16e3-328f-4143-9d0e-200121d00c4d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.192832 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.192791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9tv\" (UniqueName: \"kubernetes.io/projected/b5ed16e3-328f-4143-9d0e-200121d00c4d-kube-api-access-lg9tv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fvp57\" (UID: \"b5ed16e3-328f-4143-9d0e-200121d00c4d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.252775 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.252744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:28.523927 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:28.523903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"]
Sep 11 18:28:28.525873 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:28:28.525844    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5ed16e3_328f_4143_9d0e_200121d00c4d.slice/crio-84a2d811018673c82eb8dffbee47f3ed896e8b1d2eed5779acb3156dde7afa0f WatchSource:0}: Error finding container 84a2d811018673c82eb8dffbee47f3ed896e8b1d2eed5779acb3156dde7afa0f: Status 404 returned error can't find the container with id 84a2d811018673c82eb8dffbee47f3ed896e8b1d2eed5779acb3156dde7afa0f
Sep 11 18:28:29.294983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:29.294949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57" event={"ID":"b5ed16e3-328f-4143-9d0e-200121d00c4d","Type":"ContainerStarted","Data":"13d4e3f252876e195c007ac9474770007b0a29dc8a4bf455fb50b5bfca53844e"}
Sep 11 18:28:29.295314 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:29.294990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57" event={"ID":"b5ed16e3-328f-4143-9d0e-200121d00c4d","Type":"ContainerStarted","Data":"84a2d811018673c82eb8dffbee47f3ed896e8b1d2eed5779acb3156dde7afa0f"}
Sep 11 18:28:29.311954 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:29.311910    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57" podStartSLOduration=2.127116046 podStartE2EDuration="2.311895397s" totalImagesPullingTime="184.779351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:29.310863694 +0000 UTC m=+764.705378730" watchObservedRunningTime="2026-09-11 18:28:29.311895397 +0000 UTC m=+764.706410464"
Sep 11 18:28:30.155618 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:30.155584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:28:33.058454 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:33.058422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:33.058825 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:33.058477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:33.070709 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:33.070685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:33.326074 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:33.325997    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jsshp"
Sep 11 18:28:38.253075 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:38.253043    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:38.253075 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:38.253084    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:38.265144 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:38.265120    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:28:38.345851 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:28:38.345830    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fvp57"
Sep 11 18:29:31.183753 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:31.183718    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76687479d9-nl2b9"]
Sep 11 18:29:31.187055 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:31.187040    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76687479d9-nl2b9"
Sep 11 18:29:31.200231 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:31.200208    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76687479d9-nl2b9"]
Sep 11 18:29:31.252233 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:31.252205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c98f310-1de5-4352-8d5f-41d014340277-tls-cert\") pod \"authorino-76687479d9-nl2b9\" (UID: \"3c98f310-1de5-4352-8d5f-41d014340277\") " pod="kuadrant-system/authorino-76687479d9-nl2b9"
Sep 11 18:29:31.252337 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:31.252243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzpqh\" (UniqueName: \"kubernetes.io/projected/3c98f310-1de5-4352-8d5f-41d014340277-kube-api-access-zzpqh\") pod \"authorino-76687479d9-nl2b9\" (UID: \"3c98f310-1de5-4352-8d5f-41d014340277\") " pod="kuadrant-system/authorino-76687479d9-nl2b9"
Sep 11 18:29:31.353018 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:31.352996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c98f310-1de5-4352-8d5f-41d014340277-tls-cert\") pod \"authorino-76687479d9-nl2b9\" (UID: \"3c98f310-1de5-4352-8d5f-41d014340277\") " pod="kuadrant-system/authorino-76687479d9-nl2b9"
Sep 11 18:29:31.353133 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:31.353040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpqh\" (UniqueName: \"kubernetes.io/projected/3c98f310-1de5-4352-8d5f-41d014340277-kube-api-access-zzpqh\") pod \"authorino-76687479d9-nl2b9\" (UID: \"3c98f310-1de5-4352-8d5f-41d014340277\") " pod="kuadrant-system/authorino-76687479d9-nl2b9"
Sep 11 18:29:31.355493 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:31.355472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c98f310-1de5-4352-8d5f-41d014340277-tls-cert\") pod \"authorino-76687479d9-nl2b9\" (UID: \"3c98f310-1de5-4352-8d5f-41d014340277\") " pod="kuadrant-system/authorino-76687479d9-nl2b9"
Sep 11 18:29:31.362956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:31.362933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzpqh\" (UniqueName: \"kubernetes.io/projected/3c98f310-1de5-4352-8d5f-41d014340277-kube-api-access-zzpqh\") pod \"authorino-76687479d9-nl2b9\" (UID: \"3c98f310-1de5-4352-8d5f-41d014340277\") " pod="kuadrant-system/authorino-76687479d9-nl2b9"
Sep 11 18:29:31.495610 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:31.495563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76687479d9-nl2b9"
Sep 11 18:29:31.766496 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:31.766474    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76687479d9-nl2b9"]
Sep 11 18:29:31.768561 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:29:31.768531    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c98f310_1de5_4352_8d5f_41d014340277.slice/crio-e175fb0d70e37b1006a43c7a34fa7784f86adbc921190bf41bf5fbe878673704 WatchSource:0}: Error finding container e175fb0d70e37b1006a43c7a34fa7784f86adbc921190bf41bf5fbe878673704: Status 404 returned error can't find the container with id e175fb0d70e37b1006a43c7a34fa7784f86adbc921190bf41bf5fbe878673704
Sep 11 18:29:32.589220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:32.589181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76687479d9-nl2b9" event={"ID":"3c98f310-1de5-4352-8d5f-41d014340277","Type":"ContainerStarted","Data":"3fe600e123d3b7640e34cb5861f4552f7ceef7d450aabcaf1442fa855dc60c61"}
Sep 11 18:29:32.589646 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:32.589229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76687479d9-nl2b9" event={"ID":"3c98f310-1de5-4352-8d5f-41d014340277","Type":"ContainerStarted","Data":"e175fb0d70e37b1006a43c7a34fa7784f86adbc921190bf41bf5fbe878673704"}
Sep 11 18:29:32.608827 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:32.608775    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76687479d9-nl2b9" podStartSLOduration=0.937852993 podStartE2EDuration="1.608759503s" totalImagesPullingTime="670.90651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:32.607070106 +0000 UTC m=+828.001585134" watchObservedRunningTime="2026-09-11 18:29:32.608759503 +0000 UTC m=+828.003274526"
Sep 11 18:29:32.641747 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:32.641719    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dd9fcb464-4thh8"]
Sep 11 18:29:32.641940 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:32.641917    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dd9fcb464-4thh8" podUID="81951027-c972-4740-a697-55a6faae04bc" containerName="authorino" containerID="cri-o://082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c" gracePeriod=30
Sep 11 18:29:32.884144 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:32.884118    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dd9fcb464-4thh8"
Sep 11 18:29:32.965574 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:32.965540    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbcbm\" (UniqueName: \"kubernetes.io/projected/81951027-c972-4740-a697-55a6faae04bc-kube-api-access-kbcbm\") pod \"81951027-c972-4740-a697-55a6faae04bc\" (UID: \"81951027-c972-4740-a697-55a6faae04bc\") "
Sep 11 18:29:32.965729 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:32.965713    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81951027-c972-4740-a697-55a6faae04bc-tls-cert\") pod \"81951027-c972-4740-a697-55a6faae04bc\" (UID: \"81951027-c972-4740-a697-55a6faae04bc\") "
Sep 11 18:29:32.967732 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:32.967695    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81951027-c972-4740-a697-55a6faae04bc-kube-api-access-kbcbm" (OuterVolumeSpecName: "kube-api-access-kbcbm") pod "81951027-c972-4740-a697-55a6faae04bc" (UID: "81951027-c972-4740-a697-55a6faae04bc"). InnerVolumeSpecName "kube-api-access-kbcbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:32.976108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:32.976074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81951027-c972-4740-a697-55a6faae04bc-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "81951027-c972-4740-a697-55a6faae04bc" (UID: "81951027-c972-4740-a697-55a6faae04bc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:29:33.067363 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:33.067286    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81951027-c972-4740-a697-55a6faae04bc-tls-cert\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:29:33.067363 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:33.067316    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbcbm\" (UniqueName: \"kubernetes.io/projected/81951027-c972-4740-a697-55a6faae04bc-kube-api-access-kbcbm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:29:33.595700 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:33.595664    2563 generic.go:356] "Generic (PLEG): container finished" podID="81951027-c972-4740-a697-55a6faae04bc" containerID="082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c" exitCode=0
Sep 11 18:29:33.595700 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:33.595702    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dd9fcb464-4thh8"
Sep 11 18:29:33.596128 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:33.595748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dd9fcb464-4thh8" event={"ID":"81951027-c972-4740-a697-55a6faae04bc","Type":"ContainerDied","Data":"082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c"}
Sep 11 18:29:33.596128 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:33.595794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dd9fcb464-4thh8" event={"ID":"81951027-c972-4740-a697-55a6faae04bc","Type":"ContainerDied","Data":"34d461ddcb0899e94daaef0b21d7917b34dbb147cc4a793cb6aa2e8740dd15d4"}
Sep 11 18:29:33.596128 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:33.595816    2563 scope.go:117] "RemoveContainer" containerID="082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c"
Sep 11 18:29:33.604724 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:33.604708    2563 scope.go:117] "RemoveContainer" containerID="082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c"
Sep 11 18:29:33.604998 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:29:33.604974    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c\": container with ID starting with 082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c not found: ID does not exist" containerID="082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c"
Sep 11 18:29:33.605046 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:33.604996    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c"} err="failed to get container status \"082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c\": rpc error: code = NotFound desc = could not find container \"082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c\": container with ID starting with 082e7a252156f3a1466ffbf51252c6d86bdb19d9b248e2ba1477ffe1da6e275c not found: ID does not exist"
Sep 11 18:29:33.612817 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:33.612793    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dd9fcb464-4thh8"]
Sep 11 18:29:33.616232 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:33.616211    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dd9fcb464-4thh8"]
Sep 11 18:29:35.112260 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:35.112212    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81951027-c972-4740-a697-55a6faae04bc" path="/var/lib/kubelet/pods/81951027-c972-4740-a697-55a6faae04bc/volumes"
Sep 11 18:29:55.933855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:29:55.933820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:30:00.159022 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:00.158975    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw"]
Sep 11 18:30:00.159493 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:00.159473    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81951027-c972-4740-a697-55a6faae04bc" containerName="authorino"
Sep 11 18:30:00.159493 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:00.159491    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81951027-c972-4740-a697-55a6faae04bc" containerName="authorino"
Sep 11 18:30:00.159572 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:00.159556    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81951027-c972-4740-a697-55a6faae04bc" containerName="authorino"
Sep 11 18:30:00.163969 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:00.163952    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw"
Sep 11 18:30:00.166684 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:00.166662    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:30:00.172834 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:00.172819    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw"
Sep 11 18:30:00.176269 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:00.176251    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw"]
Sep 11 18:30:00.448523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:00.448500    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw"]
Sep 11 18:30:00.450536 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:30:00.450502    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56eebf66_9d7a_44ac_a4f4_e7d4c3cdf6da.slice/crio-9b2d70fc8f16eb56a12a8ea807349bfd47f79a040f8ac5b1f51a3f2d9b379a7a WatchSource:0}: Error finding container 9b2d70fc8f16eb56a12a8ea807349bfd47f79a040f8ac5b1f51a3f2d9b379a7a: Status 404 returned error can't find the container with id 9b2d70fc8f16eb56a12a8ea807349bfd47f79a040f8ac5b1f51a3f2d9b379a7a
Sep 11 18:30:00.722914 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:00.722844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw" event={"ID":"56eebf66-9d7a-44ac-a4f4-e7d4c3cdf6da","Type":"ContainerStarted","Data":"9b2d70fc8f16eb56a12a8ea807349bfd47f79a040f8ac5b1f51a3f2d9b379a7a"}
Sep 11 18:30:02.647151 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.647122    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"]
Sep 11 18:30:02.650846 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.650825    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:02.660082 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.660058    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:30:02.661437 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.661415    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:30:02.662039 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.661776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-52b585-serving-cert\""
Sep 11 18:30:02.662411 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.662255    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lvtx6\""
Sep 11 18:30:02.662712 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.662543    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:30:02.669297 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.669237    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"]
Sep 11 18:30:02.799037 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.799000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk9xp\" (UniqueName: \"kubernetes.io/projected/55015ade-2977-4185-a02b-f05c8364305d-kube-api-access-hk9xp\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:02.799191 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.799105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-maas-api-tls\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:02.799191 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.799146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55015ade-2977-4185-a02b-f05c8364305d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:02.799191 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.799186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-metrics-certs\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:02.899973 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.899893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-maas-api-tls\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:02.899973 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.899945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55015ade-2977-4185-a02b-f05c8364305d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:02.900219 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.899975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-metrics-certs\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:02.900219 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.900043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9xp\" (UniqueName: \"kubernetes.io/projected/55015ade-2977-4185-a02b-f05c8364305d-kube-api-access-hk9xp\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:02.900219 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:30:02.900047    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-serving-cert: secret "maas-api-e2e-inf-a-52b585-serving-cert" not found
Sep 11 18:30:02.900219 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:30:02.900143    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-maas-api-tls podName:55015ade-2977-4185-a02b-f05c8364305d nodeName:}" failed. No retries permitted until 2026-09-11 18:30:03.400122926 +0000 UTC m=+858.794637945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-maas-api-tls") pod "maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx" (UID: "55015ade-2977-4185-a02b-f05c8364305d") : secret "maas-api-e2e-inf-a-52b585-serving-cert" not found
Sep 11 18:30:02.900843 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.900786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55015ade-2977-4185-a02b-f05c8364305d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:02.902948 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.902925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-metrics-certs\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:02.920119 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:02.920086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9xp\" (UniqueName: \"kubernetes.io/projected/55015ade-2977-4185-a02b-f05c8364305d-kube-api-access-hk9xp\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:03.404855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:03.404750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-maas-api-tls\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:03.407557 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:03.407525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-maas-api-tls\") pod \"maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:03.566270 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:03.566234    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:03.980727 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:03.980694    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"]
Sep 11 18:30:03.981700 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:30:03.981668    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55015ade_2977_4185_a02b_f05c8364305d.slice/crio-5f3a8dc1bdefa36ec2a38282ac32d9fdbf27218a48f382353ea78d2de5cf2496 WatchSource:0}: Error finding container 5f3a8dc1bdefa36ec2a38282ac32d9fdbf27218a48f382353ea78d2de5cf2496: Status 404 returned error can't find the container with id 5f3a8dc1bdefa36ec2a38282ac32d9fdbf27218a48f382353ea78d2de5cf2496
Sep 11 18:30:04.036562 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.036514    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"]
Sep 11 18:30:04.041255 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.041235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:30:04.044331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.044305    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-52b585\""
Sep 11 18:30:04.044453 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.044441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-52b585-dockercfg-w8xnx\""
Sep 11 18:30:04.052311 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.052291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"]
Sep 11 18:30:04.111246 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.111217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d5gd\" (UniqueName: \"kubernetes.io/projected/54590d25-3d7a-4623-bbcf-9921dd4209ba-kube-api-access-9d5gd\") pod \"payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk\" (UID: \"54590d25-3d7a-4623-bbcf-9921dd4209ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:30:04.111398 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.111326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54590d25-3d7a-4623-bbcf-9921dd4209ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk\" (UID: \"54590d25-3d7a-4623-bbcf-9921dd4209ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:30:04.212871 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.212839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54590d25-3d7a-4623-bbcf-9921dd4209ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk\" (UID: \"54590d25-3d7a-4623-bbcf-9921dd4209ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:30:04.213026 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.212900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d5gd\" (UniqueName: \"kubernetes.io/projected/54590d25-3d7a-4623-bbcf-9921dd4209ba-kube-api-access-9d5gd\") pod \"payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk\" (UID: \"54590d25-3d7a-4623-bbcf-9921dd4209ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:30:04.213582 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.213558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54590d25-3d7a-4623-bbcf-9921dd4209ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk\" (UID: \"54590d25-3d7a-4623-bbcf-9921dd4209ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:30:04.222429 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.222334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d5gd\" (UniqueName: \"kubernetes.io/projected/54590d25-3d7a-4623-bbcf-9921dd4209ba-kube-api-access-9d5gd\") pod \"payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk\" (UID: \"54590d25-3d7a-4623-bbcf-9921dd4209ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:30:04.354419 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.354347    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:30:04.648366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.648336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"]
Sep 11 18:30:04.651120 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:30:04.651029    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54590d25_3d7a_4623_bbcf_9921dd4209ba.slice/crio-be782b7b95eaecaffadfed83189513292a332151d3b1da631655cc6475b6638a WatchSource:0}: Error finding container be782b7b95eaecaffadfed83189513292a332151d3b1da631655cc6475b6638a: Status 404 returned error can't find the container with id be782b7b95eaecaffadfed83189513292a332151d3b1da631655cc6475b6638a
Sep 11 18:30:04.755067 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.754987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk" event={"ID":"54590d25-3d7a-4623-bbcf-9921dd4209ba","Type":"ContainerStarted","Data":"be782b7b95eaecaffadfed83189513292a332151d3b1da631655cc6475b6638a"}
Sep 11 18:30:04.757113 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.757079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx" event={"ID":"55015ade-2977-4185-a02b-f05c8364305d","Type":"ContainerStarted","Data":"5f3a8dc1bdefa36ec2a38282ac32d9fdbf27218a48f382353ea78d2de5cf2496"}
Sep 11 18:30:04.759843 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.759786    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw" event={"ID":"56eebf66-9d7a-44ac-a4f4-e7d4c3cdf6da","Type":"ContainerStarted","Data":"b959160e486896e19e20d40ec678f9ff4bc1ddb8336da7002460c7110cef0f44"}
Sep 11 18:30:04.780576 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:04.780477    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw" podStartSLOduration=1.462025393 podStartE2EDuration="4.780459045s" totalImagesPullingTime="3.318433652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:04.779093172 +0000 UTC m=+860.173608223" watchObservedRunningTime="2026-09-11 18:30:04.780459045 +0000 UTC m=+860.174974073"
Sep 11 18:30:05.769671 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:05.769630    2563 generic.go:356] "Generic (PLEG): container finished" podID="56eebf66-9d7a-44ac-a4f4-e7d4c3cdf6da" containerID="b959160e486896e19e20d40ec678f9ff4bc1ddb8336da7002460c7110cef0f44" exitCode=0
Sep 11 18:30:05.770085 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:05.769707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw" event={"ID":"56eebf66-9d7a-44ac-a4f4-e7d4c3cdf6da","Type":"ContainerDied","Data":"b959160e486896e19e20d40ec678f9ff4bc1ddb8336da7002460c7110cef0f44"}
Sep 11 18:30:05.872462 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:05.872426    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"]
Sep 11 18:30:05.886980 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:05.886535    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:05.889088 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:05.889055    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"]
Sep 11 18:30:05.891188 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:05.890997    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-71b1c2-serving-cert\""
Sep 11 18:30:06.040043 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.039777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkcrb\" (UniqueName: \"kubernetes.io/projected/f5de2750-f0e7-42da-b284-3ff9bc1576ce-kube-api-access-dkcrb\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.040043 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.039951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-maas-api-tls\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.040043 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.039989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-metrics-certs\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.040043 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.040013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5de2750-f0e7-42da-b284-3ff9bc1576ce-openshift-service-ca\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.140735 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.140701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-maas-api-tls\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.140905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.140744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-metrics-certs\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.140905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.140770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5de2750-f0e7-42da-b284-3ff9bc1576ce-openshift-service-ca\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.140905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.140806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkcrb\" (UniqueName: \"kubernetes.io/projected/f5de2750-f0e7-42da-b284-3ff9bc1576ce-kube-api-access-dkcrb\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.141860 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.141833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5de2750-f0e7-42da-b284-3ff9bc1576ce-openshift-service-ca\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.144005 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.143962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-metrics-certs\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.144127 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.144092    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-maas-api-tls\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.155906 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.155886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkcrb\" (UniqueName: \"kubernetes.io/projected/f5de2750-f0e7-42da-b284-3ff9bc1576ce-kube-api-access-dkcrb\") pod \"maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:06.203948 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:06.203920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:07.084192 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.084114    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"]
Sep 11 18:30:07.088230 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.088207    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:30:07.091035 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.091018    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-71b1c2-dockercfg-xbrm2\""
Sep 11 18:30:07.091302 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.091286    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-71b1c2\""
Sep 11 18:30:07.103483 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.103464    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"]
Sep 11 18:30:07.135990 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.135804    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw"
Sep 11 18:30:07.252080 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.252039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffcf6b9-3369-4fd9-a942-82cf812a10f8-plugins-config-volume\") pod \"payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg\" (UID: \"fffcf6b9-3369-4fd9-a942-82cf812a10f8\") " pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:30:07.252260 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.252160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lj6k\" (UniqueName: \"kubernetes.io/projected/fffcf6b9-3369-4fd9-a942-82cf812a10f8-kube-api-access-8lj6k\") pod \"payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg\" (UID: \"fffcf6b9-3369-4fd9-a942-82cf812a10f8\") " pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:30:07.353046 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.352972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lj6k\" (UniqueName: \"kubernetes.io/projected/fffcf6b9-3369-4fd9-a942-82cf812a10f8-kube-api-access-8lj6k\") pod \"payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg\" (UID: \"fffcf6b9-3369-4fd9-a942-82cf812a10f8\") " pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:30:07.353188 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.353086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffcf6b9-3369-4fd9-a942-82cf812a10f8-plugins-config-volume\") pod \"payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg\" (UID: \"fffcf6b9-3369-4fd9-a942-82cf812a10f8\") " pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:30:07.353808 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.353789    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffcf6b9-3369-4fd9-a942-82cf812a10f8-plugins-config-volume\") pod \"payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg\" (UID: \"fffcf6b9-3369-4fd9-a942-82cf812a10f8\") " pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:30:07.376330 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.376305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lj6k\" (UniqueName: \"kubernetes.io/projected/fffcf6b9-3369-4fd9-a942-82cf812a10f8-kube-api-access-8lj6k\") pod \"payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg\" (UID: \"fffcf6b9-3369-4fd9-a942-82cf812a10f8\") " pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:30:07.399150 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.399128    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:30:07.729099 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:30:07.729070    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5de2750_f0e7_42da_b284_3ff9bc1576ce.slice/crio-a62dffb359a4aa248f03c2563d608f518ad83c332e62316b205da0b0d042b37e WatchSource:0}: Error finding container a62dffb359a4aa248f03c2563d608f518ad83c332e62316b205da0b0d042b37e: Status 404 returned error can't find the container with id a62dffb359a4aa248f03c2563d608f518ad83c332e62316b205da0b0d042b37e
Sep 11 18:30:07.729332 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.729314    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"]
Sep 11 18:30:07.729790 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:30:07.729711    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffcf6b9_3369_4fd9_a942_82cf812a10f8.slice/crio-2e26ea1e8cde723f490598c5729dc0b91d4fe64cdd8d723e61ef62b6fb387f9f WatchSource:0}: Error finding container 2e26ea1e8cde723f490598c5729dc0b91d4fe64cdd8d723e61ef62b6fb387f9f: Status 404 returned error can't find the container with id 2e26ea1e8cde723f490598c5729dc0b91d4fe64cdd8d723e61ef62b6fb387f9f
Sep 11 18:30:07.731333 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.731305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"]
Sep 11 18:30:07.780156 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.780131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r" event={"ID":"f5de2750-f0e7-42da-b284-3ff9bc1576ce","Type":"ContainerStarted","Data":"a62dffb359a4aa248f03c2563d608f518ad83c332e62316b205da0b0d042b37e"}
Sep 11 18:30:07.781583 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.781561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx" event={"ID":"55015ade-2977-4185-a02b-f05c8364305d","Type":"ContainerStarted","Data":"cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3"}
Sep 11 18:30:07.781767 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.781690    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:07.783122 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.783096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg" event={"ID":"fffcf6b9-3369-4fd9-a942-82cf812a10f8","Type":"ContainerStarted","Data":"2e26ea1e8cde723f490598c5729dc0b91d4fe64cdd8d723e61ef62b6fb387f9f"}
Sep 11 18:30:07.784594 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.784568    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw"
Sep 11 18:30:07.784692 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.784606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-dkgpw" event={"ID":"56eebf66-9d7a-44ac-a4f4-e7d4c3cdf6da","Type":"ContainerDied","Data":"9b2d70fc8f16eb56a12a8ea807349bfd47f79a040f8ac5b1f51a3f2d9b379a7a"}
Sep 11 18:30:07.784692 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.784648    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b2d70fc8f16eb56a12a8ea807349bfd47f79a040f8ac5b1f51a3f2d9b379a7a"
Sep 11 18:30:07.786366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.786242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk" event={"ID":"54590d25-3d7a-4623-bbcf-9921dd4209ba","Type":"ContainerStarted","Data":"a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217"}
Sep 11 18:30:07.786366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.786297    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:30:07.824324 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.824280    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx" podStartSLOduration=2.368992837 podStartE2EDuration="5.824266103s" totalImagesPullingTime="3.455273266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:07.822598411 +0000 UTC m=+863.217113439" watchObservedRunningTime="2026-09-11 18:30:07.824266103 +0000 UTC m=+863.218781126"
Sep 11 18:30:07.857599 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:07.857549    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk" podStartSLOduration=1.023644219 podStartE2EDuration="3.857531056s" totalImagesPullingTime="2.833886837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:07.855582812 +0000 UTC m=+863.250097857" watchObservedRunningTime="2026-09-11 18:30:07.857531056 +0000 UTC m=+863.252046082"
Sep 11 18:30:08.793666 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:08.793630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r" event={"ID":"f5de2750-f0e7-42da-b284-3ff9bc1576ce","Type":"ContainerStarted","Data":"afb513e2f7f45a172a6bde42e7e48c5f9b6c402bce8ff347a044453738344f38"}
Sep 11 18:30:08.794076 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:08.793743    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:08.795331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:08.795299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg" event={"ID":"fffcf6b9-3369-4fd9-a942-82cf812a10f8","Type":"ContainerStarted","Data":"b689d2266ec798b95da194f90b2017fb9f44a0e727ecbe1e2627112ee39f8ae8"}
Sep 11 18:30:08.795486 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:08.795356    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:30:08.820879 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:08.820836    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r" podStartSLOduration=3.549886807 podStartE2EDuration="3.820823631s" totalImagesPullingTime="270.936824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:08.819225828 +0000 UTC m=+864.213740885" watchObservedRunningTime="2026-09-11 18:30:08.820823631 +0000 UTC m=+864.215338656"
Sep 11 18:30:08.845096 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:08.845052    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg" podStartSLOduration=1.845036839 podStartE2EDuration="1.845036839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:08.844326591 +0000 UTC m=+864.238841634" watchObservedRunningTime="2026-09-11 18:30:08.845036839 +0000 UTC m=+864.239551863"
Sep 11 18:30:12.766384 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:12.766336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:30:13.801083 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:13.801048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:30:14.814070 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:14.814040    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:30:18.797400 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:18.797355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:30:19.802948 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:19.802921    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:30:31.679400 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:31.679353    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:30:35.423625 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.423588    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"]
Sep 11 18:30:35.424165 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.424151    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56eebf66-9d7a-44ac-a4f4-e7d4c3cdf6da" containerName="cleanup"
Sep 11 18:30:35.424213 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.424168    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56eebf66-9d7a-44ac-a4f4-e7d4c3cdf6da" containerName="cleanup"
Sep 11 18:30:35.424261 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.424251    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56eebf66-9d7a-44ac-a4f4-e7d4c3cdf6da" containerName="cleanup"
Sep 11 18:30:35.494555 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.494517    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"]
Sep 11 18:30:35.494709 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.494614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.497571 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.497550    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0d18cc-serving-cert\""
Sep 11 18:30:35.578507 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.578485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-metrics-certs\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.578573 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.578524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660777a4-f485-4d3f-9fd3-20f355cbbc11-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.578662 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.578637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.578774 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.578753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ztr\" (UniqueName: \"kubernetes.io/projected/660777a4-f485-4d3f-9fd3-20f355cbbc11-kube-api-access-v6ztr\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.679977 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.679919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.680084 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.679979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ztr\" (UniqueName: \"kubernetes.io/projected/660777a4-f485-4d3f-9fd3-20f355cbbc11-kube-api-access-v6ztr\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.680084 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.680001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-metrics-certs\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.680084 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.680017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660777a4-f485-4d3f-9fd3-20f355cbbc11-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.680635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.680617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660777a4-f485-4d3f-9fd3-20f355cbbc11-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.682161 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.682141    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.682246 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.682223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-metrics-certs\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.690851 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.690830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ztr\" (UniqueName: \"kubernetes.io/projected/660777a4-f485-4d3f-9fd3-20f355cbbc11-kube-api-access-v6ztr\") pod \"maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:35.836639 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:35.836618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:36.114326 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.114304    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"]
Sep 11 18:30:36.115849 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:30:36.115822    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660777a4_f485_4d3f_9fd3_20f355cbbc11.slice/crio-7005fee7b6b8274ec545eecb2e7ea0f5e47e1f4dae746c325841002e67fc9f8e WatchSource:0}: Error finding container 7005fee7b6b8274ec545eecb2e7ea0f5e47e1f4dae746c325841002e67fc9f8e: Status 404 returned error can't find the container with id 7005fee7b6b8274ec545eecb2e7ea0f5e47e1f4dae746c325841002e67fc9f8e
Sep 11 18:30:36.253111 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.253067    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"]
Sep 11 18:30:36.255727 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.255704    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:30:36.259737 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.259714    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"]
Sep 11 18:30:36.260412 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.260386    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0d18cc\""
Sep 11 18:30:36.262230 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.262196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:30:36.272518 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.272496    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"]
Sep 11 18:30:36.277577 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.277558    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"]
Sep 11 18:30:36.285796 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.285771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kxz4\" (UniqueName: \"kubernetes.io/projected/a2b21d25-e7d9-4022-9092-d30b648eace1-kube-api-access-4kxz4\") pod \"payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk\" (UID: \"a2b21d25-e7d9-4022-9092-d30b648eace1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:30:36.285881 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.285811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4zkk\" (UniqueName: \"kubernetes.io/projected/6440c884-b9ad-423a-a548-371ede8cdb3c-kube-api-access-h4zkk\") pod \"payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d\" (UID: \"6440c884-b9ad-423a-a548-371ede8cdb3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:30:36.285940 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.285879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b21d25-e7d9-4022-9092-d30b648eace1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk\" (UID: \"a2b21d25-e7d9-4022-9092-d30b648eace1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:30:36.286016 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.286000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6440c884-b9ad-423a-a548-371ede8cdb3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d\" (UID: \"6440c884-b9ad-423a-a548-371ede8cdb3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:30:36.387451 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.387292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kxz4\" (UniqueName: \"kubernetes.io/projected/a2b21d25-e7d9-4022-9092-d30b648eace1-kube-api-access-4kxz4\") pod \"payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk\" (UID: \"a2b21d25-e7d9-4022-9092-d30b648eace1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:30:36.387451 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.387429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zkk\" (UniqueName: \"kubernetes.io/projected/6440c884-b9ad-423a-a548-371ede8cdb3c-kube-api-access-h4zkk\") pod \"payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d\" (UID: \"6440c884-b9ad-423a-a548-371ede8cdb3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:30:36.387609 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.387493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b21d25-e7d9-4022-9092-d30b648eace1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk\" (UID: \"a2b21d25-e7d9-4022-9092-d30b648eace1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:30:36.387609 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.387591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6440c884-b9ad-423a-a548-371ede8cdb3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d\" (UID: \"6440c884-b9ad-423a-a548-371ede8cdb3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:30:36.388242 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.388220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6440c884-b9ad-423a-a548-371ede8cdb3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d\" (UID: \"6440c884-b9ad-423a-a548-371ede8cdb3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:30:36.388242 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.388235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b21d25-e7d9-4022-9092-d30b648eace1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk\" (UID: \"a2b21d25-e7d9-4022-9092-d30b648eace1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:30:36.395494 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.395464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4zkk\" (UniqueName: \"kubernetes.io/projected/6440c884-b9ad-423a-a548-371ede8cdb3c-kube-api-access-h4zkk\") pod \"payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d\" (UID: \"6440c884-b9ad-423a-a548-371ede8cdb3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:30:36.395573 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.395503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kxz4\" (UniqueName: \"kubernetes.io/projected/a2b21d25-e7d9-4022-9092-d30b648eace1-kube-api-access-4kxz4\") pod \"payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk\" (UID: \"a2b21d25-e7d9-4022-9092-d30b648eace1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:30:36.569452 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.569436    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:30:36.575737 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.575716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:30:36.855748 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.855724    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"]
Sep 11 18:30:36.857224 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:30:36.857191    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6440c884_b9ad_423a_a548_371ede8cdb3c.slice/crio-9f74dc10bf0bd86ca8d65c51dd7365820c703c86a2a6c0dc5c9d9f430884639c WatchSource:0}: Error finding container 9f74dc10bf0bd86ca8d65c51dd7365820c703c86a2a6c0dc5c9d9f430884639c: Status 404 returned error can't find the container with id 9f74dc10bf0bd86ca8d65c51dd7365820c703c86a2a6c0dc5c9d9f430884639c
Sep 11 18:30:36.862173 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.862070    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"]
Sep 11 18:30:36.865839 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:30:36.865815    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b21d25_e7d9_4022_9092_d30b648eace1.slice/crio-c711d97e8a753efa3935db514d7e32fd89b1ffd6e88bbbca88cc3a9200f71b96 WatchSource:0}: Error finding container c711d97e8a753efa3935db514d7e32fd89b1ffd6e88bbbca88cc3a9200f71b96: Status 404 returned error can't find the container with id c711d97e8a753efa3935db514d7e32fd89b1ffd6e88bbbca88cc3a9200f71b96
Sep 11 18:30:36.939919 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.939886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk" event={"ID":"a2b21d25-e7d9-4022-9092-d30b648eace1","Type":"ContainerStarted","Data":"5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60"}
Sep 11 18:30:36.940092 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.939961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk" event={"ID":"a2b21d25-e7d9-4022-9092-d30b648eace1","Type":"ContainerStarted","Data":"c711d97e8a753efa3935db514d7e32fd89b1ffd6e88bbbca88cc3a9200f71b96"}
Sep 11 18:30:36.940156 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.940086    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:30:36.941998 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.941971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d" event={"ID":"6440c884-b9ad-423a-a548-371ede8cdb3c","Type":"ContainerStarted","Data":"ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68"}
Sep 11 18:30:36.942118 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.942011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d" event={"ID":"6440c884-b9ad-423a-a548-371ede8cdb3c","Type":"ContainerStarted","Data":"9f74dc10bf0bd86ca8d65c51dd7365820c703c86a2a6c0dc5c9d9f430884639c"}
Sep 11 18:30:36.942118 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.942037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:30:36.943831 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.943808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr" event={"ID":"660777a4-f485-4d3f-9fd3-20f355cbbc11","Type":"ContainerStarted","Data":"e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7"}
Sep 11 18:30:36.943940 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.943850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr" event={"ID":"660777a4-f485-4d3f-9fd3-20f355cbbc11","Type":"ContainerStarted","Data":"7005fee7b6b8274ec545eecb2e7ea0f5e47e1f4dae746c325841002e67fc9f8e"}
Sep 11 18:30:36.943940 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.943873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:36.965448 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.965343    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk" podStartSLOduration=0.965323274 podStartE2EDuration="965.323274ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:36.964697914 +0000 UTC m=+892.359212955" watchObservedRunningTime="2026-09-11 18:30:36.965323274 +0000 UTC m=+892.359838298"
Sep 11 18:30:36.986899 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:36.986600    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr" podStartSLOduration=1.607517746 podStartE2EDuration="1.986582164s" totalImagesPullingTime="379.064418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:36.986171002 +0000 UTC m=+892.380686236" watchObservedRunningTime="2026-09-11 18:30:36.986582164 +0000 UTC m=+892.381097191"
Sep 11 18:30:37.010034 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:37.009991    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d" podStartSLOduration=1.009977563 podStartE2EDuration="1.009977563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:37.007969699 +0000 UTC m=+892.402484725" watchObservedRunningTime="2026-09-11 18:30:37.009977563 +0000 UTC m=+892.404492586"
Sep 11 18:30:42.955507 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:42.955434    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:30:42.965769 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:42.965743    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:30:43.275212 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.275126    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"]
Sep 11 18:30:43.277573 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.277557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.281528 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.281507    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b63ece\"/\"default-dockercfg-nbccq\""
Sep 11 18:30:43.281705 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.281663    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b63ece\"/\"simulator-w0-b63ece-kserve-self-signed-certs\""
Sep 11 18:30:43.281771 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.281722    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b63ece\"/\"openshift-service-ca.crt\""
Sep 11 18:30:43.281771 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.281752    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b63ece\"/\"kube-root-ca.crt\""
Sep 11 18:30:43.300482 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.300460    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"]
Sep 11 18:30:43.344155 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.344120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-dshm\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.344155 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.344155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-tmp-dir\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.344332 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.344184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12878d5b-505b-4540-a7c4-2709ac559667-tls-certs\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.344332 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.344279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfstx\" (UniqueName: \"kubernetes.io/projected/12878d5b-505b-4540-a7c4-2709ac559667-kube-api-access-qfstx\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.344411 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.344360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-model-cache\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.344445 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.344422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-home\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.445779 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.445743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfstx\" (UniqueName: \"kubernetes.io/projected/12878d5b-505b-4540-a7c4-2709ac559667-kube-api-access-qfstx\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.445946 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.445853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-model-cache\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.445946 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.445899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-home\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.446065 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.445977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-dshm\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.446065 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.446003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-tmp-dir\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.446065 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.446031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12878d5b-505b-4540-a7c4-2709ac559667-tls-certs\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.446303 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.446278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-model-cache\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.446406 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.446308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-home\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.446498 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.446407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-tmp-dir\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.448150 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.448121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-dshm\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.448698 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.448678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12878d5b-505b-4540-a7c4-2709ac559667-tls-certs\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.468431 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.468409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfstx\" (UniqueName: \"kubernetes.io/projected/12878d5b-505b-4540-a7c4-2709ac559667-kube-api-access-qfstx\") pod \"simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") " pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.587947 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.587866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:43.882254 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:30:43.882226    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12878d5b_505b_4540_a7c4_2709ac559667.slice/crio-8cb50ec9cde1dfce96544490276ba55db79f80633e3a77ffe3e2eb0b409a116b WatchSource:0}: Error finding container 8cb50ec9cde1dfce96544490276ba55db79f80633e3a77ffe3e2eb0b409a116b: Status 404 returned error can't find the container with id 8cb50ec9cde1dfce96544490276ba55db79f80633e3a77ffe3e2eb0b409a116b
Sep 11 18:30:43.891153 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.891128    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"]
Sep 11 18:30:43.980611 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.980581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7" event={"ID":"12878d5b-505b-4540-a7c4-2709ac559667","Type":"ContainerStarted","Data":"55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139"}
Sep 11 18:30:43.980910 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:43.980626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7" event={"ID":"12878d5b-505b-4540-a7c4-2709ac559667","Type":"ContainerStarted","Data":"8cb50ec9cde1dfce96544490276ba55db79f80633e3a77ffe3e2eb0b409a116b"}
Sep 11 18:30:44.022119 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:44.022078    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7" podStartSLOduration=1.022064736 podStartE2EDuration="1.022064736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:44.018356844 +0000 UTC m=+899.412871867" watchObservedRunningTime="2026-09-11 18:30:44.022064736 +0000 UTC m=+899.416579761"
Sep 11 18:30:45.087028 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:45.086998    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:30:45.088928 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:45.088898    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:30:45.094619 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:45.094600    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:30:45.096643 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:45.096624    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:30:47.950752 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:47.950721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:30:47.951145 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:47.950989    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:30:53.588432 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:53.588401    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:53.588850 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:53.588449    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:53.600869 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:53.600847    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:30:54.039151 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:30:54.039125    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:31:00.770352 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:00.770315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:31:15.580075 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:15.580041    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:31:36.960365 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:36.960331    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:31:43.634129 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.634094    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"]
Sep 11 18:31:43.636444 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.636427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.640151 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.640124    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c8b274\"/\"openshift-service-ca.crt\""
Sep 11 18:31:43.640272 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.640170    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c8b274\"/\"default-dockercfg-wnwtz\""
Sep 11 18:31:43.640272 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.640210    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c8b274\"/\"simulator-w3-c8b274-kserve-self-signed-certs\""
Sep 11 18:31:43.640475 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.640459    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c8b274\"/\"kube-root-ca.crt\""
Sep 11 18:31:43.646602 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.646583    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"]
Sep 11 18:31:43.734197 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.734178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-model-cache\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.734293 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.734205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae996dc-66d2-44b6-99c4-d66c080476f3-tls-certs\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.734293 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.734226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-home\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.734397 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.734294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-tmp-dir\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.734397 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.734360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-dshm\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.734501 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.734418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64j7\" (UniqueName: \"kubernetes.io/projected/8ae996dc-66d2-44b6-99c4-d66c080476f3-kube-api-access-d64j7\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.835584 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.835560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-model-cache\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.835684 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.835591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae996dc-66d2-44b6-99c4-d66c080476f3-tls-certs\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.835684 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.835611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-home\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.835684 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.835635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-tmp-dir\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.835842 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.835764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-dshm\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.835842 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.835814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64j7\" (UniqueName: \"kubernetes.io/projected/8ae996dc-66d2-44b6-99c4-d66c080476f3-kube-api-access-d64j7\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.836047 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.835999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-model-cache\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.836047 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.836028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-tmp-dir\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.836223 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.836103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-home\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.837985 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.837956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-dshm\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.838162 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.838145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae996dc-66d2-44b6-99c4-d66c080476f3-tls-certs\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.844818 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.844796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64j7\" (UniqueName: \"kubernetes.io/projected/8ae996dc-66d2-44b6-99c4-d66c080476f3-kube-api-access-d64j7\") pod \"simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") " pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:43.946964 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:43.946937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:44.219499 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:44.219470    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"]
Sep 11 18:31:44.221072 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:31:44.221049    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ae996dc_66d2_44b6_99c4_d66c080476f3.slice/crio-c77e652ad8f46b1b03cad84c0311ef162c6885e7284e9b79a2bdccc5ab19daa2 WatchSource:0}: Error finding container c77e652ad8f46b1b03cad84c0311ef162c6885e7284e9b79a2bdccc5ab19daa2: Status 404 returned error can't find the container with id c77e652ad8f46b1b03cad84c0311ef162c6885e7284e9b79a2bdccc5ab19daa2
Sep 11 18:31:44.281445 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:44.281409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm" event={"ID":"8ae996dc-66d2-44b6-99c4-d66c080476f3","Type":"ContainerStarted","Data":"faf8e4012737f51b5eefb620b0f90d99128027bffa6a5b922c60649a445026e0"}
Sep 11 18:31:44.281553 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:44.281465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm" event={"ID":"8ae996dc-66d2-44b6-99c4-d66c080476f3","Type":"ContainerStarted","Data":"c77e652ad8f46b1b03cad84c0311ef162c6885e7284e9b79a2bdccc5ab19daa2"}
Sep 11 18:31:44.300032 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:44.299982    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm" podStartSLOduration=1.299963519 podStartE2EDuration="1.299963519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:44.298939486 +0000 UTC m=+959.693454538" watchObservedRunningTime="2026-09-11 18:31:44.299963519 +0000 UTC m=+959.694478546"
Sep 11 18:31:48.569334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:48.569303    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:31:53.947996 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:53.947963    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:53.948396 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:53.948014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:53.960163 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:53.960141    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:31:54.342183 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:31:54.342118    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:32:06.249106 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:06.249072    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:32:27.581722 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.581689    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"]
Sep 11 18:32:27.585490 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.585470    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.588116 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.588088    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-71b1c2\"/\"default-dockercfg-js875\""
Sep 11 18:32:27.589485 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.589464    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-71b1c2\"/\"kube-root-ca.crt\""
Sep 11 18:32:27.589595 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.589487    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-71b1c2\"/\"autores-model-71b1c2-kserve-self-signed-certs\""
Sep 11 18:32:27.589595 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.589491    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-71b1c2\"/\"openshift-service-ca.crt\""
Sep 11 18:32:27.591265 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.591051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tmp-dir\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.591265 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.591102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-model-cache\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.591265 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.591132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-home\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.591265 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.591162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-dshm\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.591265 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.591187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bzgb\" (UniqueName: \"kubernetes.io/projected/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-kube-api-access-4bzgb\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.591604 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.591310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tls-certs\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.598268 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.598243    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"]
Sep 11 18:32:27.692490 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.692462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tmp-dir\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.692640 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.692518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-model-cache\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.692640 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.692540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-home\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.692640 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.692560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-dshm\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.692640 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.692577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bzgb\" (UniqueName: \"kubernetes.io/projected/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-kube-api-access-4bzgb\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.692640 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.692638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tls-certs\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.692893 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.692739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tmp-dir\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.692893 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.692823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-home\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.692978 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.692924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-model-cache\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.694686 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.694663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-dshm\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.694971 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.694952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tls-certs\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.707427 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.707406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bzgb\" (UniqueName: \"kubernetes.io/projected/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-kube-api-access-4bzgb\") pod \"autores-model-71b1c2-kserve-678d79c995-cdw65\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") " pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:27.894440 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:27.894349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:28.170006 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:28.169979    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"]
Sep 11 18:32:28.171560 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:32:28.171520    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3aa66a09_17c5_4e65_83b5_88c1987dd4e7.slice/crio-7a128eaccfb6ee5622d21913eb1ed111ee2f3eeaf8b31060c63805b8ff835856 WatchSource:0}: Error finding container 7a128eaccfb6ee5622d21913eb1ed111ee2f3eeaf8b31060c63805b8ff835856: Status 404 returned error can't find the container with id 7a128eaccfb6ee5622d21913eb1ed111ee2f3eeaf8b31060c63805b8ff835856
Sep 11 18:32:28.498206 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:28.498124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65" event={"ID":"3aa66a09-17c5-4e65-83b5-88c1987dd4e7","Type":"ContainerStarted","Data":"ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749"}
Sep 11 18:32:28.498206 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:28.498170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65" event={"ID":"3aa66a09-17c5-4e65-83b5-88c1987dd4e7","Type":"ContainerStarted","Data":"7a128eaccfb6ee5622d21913eb1ed111ee2f3eeaf8b31060c63805b8ff835856"}
Sep 11 18:32:28.518644 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:28.518603    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65" podStartSLOduration=1.518587252 podStartE2EDuration="1.518587252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:28.518152819 +0000 UTC m=+1003.912667844" watchObservedRunningTime="2026-09-11 18:32:28.518587252 +0000 UTC m=+1003.913102322"
Sep 11 18:32:33.380225 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:33.380192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:32:37.895227 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:37.895162    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:37.895227 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:37.895232    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:37.907651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:37.907629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:38.587259 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:38.587229    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:32:53.659102 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:32:53.659071    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:33:05.234399 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:05.234356    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"]
Sep 11 18:33:13.443236 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.443203    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"]
Sep 11 18:33:13.443720 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.443490    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r" podUID="f5de2750-f0e7-42da-b284-3ff9bc1576ce" containerName="maas-api" containerID="cri-o://afb513e2f7f45a172a6bde42e7e48c5f9b6c402bce8ff347a044453738344f38" gracePeriod=30
Sep 11 18:33:13.502893 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.502860    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"]
Sep 11 18:33:13.503143 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.503115    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg" podUID="fffcf6b9-3369-4fd9-a942-82cf812a10f8" containerName="payload-processing" containerID="cri-o://b689d2266ec798b95da194f90b2017fb9f44a0e727ecbe1e2627112ee39f8ae8" gracePeriod=30
Sep 11 18:33:13.747505 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.747477    2563 generic.go:356] "Generic (PLEG): container finished" podID="f5de2750-f0e7-42da-b284-3ff9bc1576ce" containerID="afb513e2f7f45a172a6bde42e7e48c5f9b6c402bce8ff347a044453738344f38" exitCode=0
Sep 11 18:33:13.747652 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.747550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r" event={"ID":"f5de2750-f0e7-42da-b284-3ff9bc1576ce","Type":"ContainerDied","Data":"afb513e2f7f45a172a6bde42e7e48c5f9b6c402bce8ff347a044453738344f38"}
Sep 11 18:33:13.749436 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.749413    2563 generic.go:356] "Generic (PLEG): container finished" podID="fffcf6b9-3369-4fd9-a942-82cf812a10f8" containerID="b689d2266ec798b95da194f90b2017fb9f44a0e727ecbe1e2627112ee39f8ae8" exitCode=0
Sep 11 18:33:13.749568 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.749492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg" event={"ID":"fffcf6b9-3369-4fd9-a942-82cf812a10f8","Type":"ContainerDied","Data":"b689d2266ec798b95da194f90b2017fb9f44a0e727ecbe1e2627112ee39f8ae8"}
Sep 11 18:33:13.768746 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.768730    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:33:13.783453 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.783435    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:33:13.872990 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.872963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-maas-api-tls\") pod \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") "
Sep 11 18:33:13.873137 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.873086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5de2750-f0e7-42da-b284-3ff9bc1576ce-openshift-service-ca\") pod \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") "
Sep 11 18:33:13.873187 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.873140    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkcrb\" (UniqueName: \"kubernetes.io/projected/f5de2750-f0e7-42da-b284-3ff9bc1576ce-kube-api-access-dkcrb\") pod \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") "
Sep 11 18:33:13.873187 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.873178    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-metrics-certs\") pod \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\" (UID: \"f5de2750-f0e7-42da-b284-3ff9bc1576ce\") "
Sep 11 18:33:13.873527 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.873499    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5de2750-f0e7-42da-b284-3ff9bc1576ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f5de2750-f0e7-42da-b284-3ff9bc1576ce" (UID: "f5de2750-f0e7-42da-b284-3ff9bc1576ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:33:13.875329 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.875301    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5de2750-f0e7-42da-b284-3ff9bc1576ce-kube-api-access-dkcrb" (OuterVolumeSpecName: "kube-api-access-dkcrb") pod "f5de2750-f0e7-42da-b284-3ff9bc1576ce" (UID: "f5de2750-f0e7-42da-b284-3ff9bc1576ce"). InnerVolumeSpecName "kube-api-access-dkcrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:33:13.875460 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.875299    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f5de2750-f0e7-42da-b284-3ff9bc1576ce" (UID: "f5de2750-f0e7-42da-b284-3ff9bc1576ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:33:13.875460 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.875398    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f5de2750-f0e7-42da-b284-3ff9bc1576ce" (UID: "f5de2750-f0e7-42da-b284-3ff9bc1576ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:33:13.974259 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.974204    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lj6k\" (UniqueName: \"kubernetes.io/projected/fffcf6b9-3369-4fd9-a942-82cf812a10f8-kube-api-access-8lj6k\") pod \"fffcf6b9-3369-4fd9-a942-82cf812a10f8\" (UID: \"fffcf6b9-3369-4fd9-a942-82cf812a10f8\") "
Sep 11 18:33:13.974259 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.974230    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffcf6b9-3369-4fd9-a942-82cf812a10f8-plugins-config-volume\") pod \"fffcf6b9-3369-4fd9-a942-82cf812a10f8\" (UID: \"fffcf6b9-3369-4fd9-a942-82cf812a10f8\") "
Sep 11 18:33:13.974526 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.974510    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5de2750-f0e7-42da-b284-3ff9bc1576ce-openshift-service-ca\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:13.974574 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.974529    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkcrb\" (UniqueName: \"kubernetes.io/projected/f5de2750-f0e7-42da-b284-3ff9bc1576ce-kube-api-access-dkcrb\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:13.974574 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.974538    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-metrics-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:13.974574 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.974546    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5de2750-f0e7-42da-b284-3ff9bc1576ce-maas-api-tls\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:13.974663 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.974617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffcf6b9-3369-4fd9-a942-82cf812a10f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fffcf6b9-3369-4fd9-a942-82cf812a10f8" (UID: "fffcf6b9-3369-4fd9-a942-82cf812a10f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:33:13.976078 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:13.976056    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffcf6b9-3369-4fd9-a942-82cf812a10f8-kube-api-access-8lj6k" (OuterVolumeSpecName: "kube-api-access-8lj6k") pod "fffcf6b9-3369-4fd9-a942-82cf812a10f8" (UID: "fffcf6b9-3369-4fd9-a942-82cf812a10f8"). InnerVolumeSpecName "kube-api-access-8lj6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:33:14.075694 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.075673    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffcf6b9-3369-4fd9-a942-82cf812a10f8-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:14.075694 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.075692    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lj6k\" (UniqueName: \"kubernetes.io/projected/fffcf6b9-3369-4fd9-a942-82cf812a10f8-kube-api-access-8lj6k\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:14.755696 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.755622    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"
Sep 11 18:33:14.755696 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.755630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r" event={"ID":"f5de2750-f0e7-42da-b284-3ff9bc1576ce","Type":"ContainerDied","Data":"a62dffb359a4aa248f03c2563d608f518ad83c332e62316b205da0b0d042b37e"}
Sep 11 18:33:14.755696 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.755684    2563 scope.go:117] "RemoveContainer" containerID="afb513e2f7f45a172a6bde42e7e48c5f9b6c402bce8ff347a044453738344f38"
Sep 11 18:33:14.757170 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.757143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg" event={"ID":"fffcf6b9-3369-4fd9-a942-82cf812a10f8","Type":"ContainerDied","Data":"2e26ea1e8cde723f490598c5729dc0b91d4fe64cdd8d723e61ef62b6fb387f9f"}
Sep 11 18:33:14.757268 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.757146    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"
Sep 11 18:33:14.766324 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.766309    2563 scope.go:117] "RemoveContainer" containerID="b689d2266ec798b95da194f90b2017fb9f44a0e727ecbe1e2627112ee39f8ae8"
Sep 11 18:33:14.780991 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.780970    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"]
Sep 11 18:33:14.784904 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.784884    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-71b1c2-7b894dc579-tbd9r"]
Sep 11 18:33:14.800057 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.800033    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"]
Sep 11 18:33:14.807775 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:14.807756    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71b1c2-87bd578c7-gd5zg"]
Sep 11 18:33:15.051797 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:15.051737    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:33:15.113167 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:15.113143    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5de2750-f0e7-42da-b284-3ff9bc1576ce" path="/var/lib/kubelet/pods/f5de2750-f0e7-42da-b284-3ff9bc1576ce/volumes"
Sep 11 18:33:15.113493 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:15.113480    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fffcf6b9-3369-4fd9-a942-82cf812a10f8" path="/var/lib/kubelet/pods/fffcf6b9-3369-4fd9-a942-82cf812a10f8/volumes"
Sep 11 18:33:20.253184 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.253135    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65" podUID="3aa66a09-17c5-4e65-83b5-88c1987dd4e7" containerName="main" containerID="cri-o://ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749" gracePeriod=45
Sep 11 18:33:20.495827 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.495805    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:33:20.529881 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.529826    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tmp-dir\") pod \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") "
Sep 11 18:33:20.529881 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.529859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-dshm\") pod \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") "
Sep 11 18:33:20.529881 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.529881    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tls-certs\") pod \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") "
Sep 11 18:33:20.530124 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.529911    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-model-cache\") pod \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") "
Sep 11 18:33:20.530124 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.529952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-home\") pod \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") "
Sep 11 18:33:20.530124 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.530040    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bzgb\" (UniqueName: \"kubernetes.io/projected/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-kube-api-access-4bzgb\") pod \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\" (UID: \"3aa66a09-17c5-4e65-83b5-88c1987dd4e7\") "
Sep 11 18:33:20.530124 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.530051    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3aa66a09-17c5-4e65-83b5-88c1987dd4e7" (UID: "3aa66a09-17c5-4e65-83b5-88c1987dd4e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:20.530337 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.530188    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "3aa66a09-17c5-4e65-83b5-88c1987dd4e7" (UID: "3aa66a09-17c5-4e65-83b5-88c1987dd4e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:20.530337 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.530268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-home" (OuterVolumeSpecName: "home") pod "3aa66a09-17c5-4e65-83b5-88c1987dd4e7" (UID: "3aa66a09-17c5-4e65-83b5-88c1987dd4e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:20.530459 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.530435    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:20.530517 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.530464    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:20.530517 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.530477    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:20.532359 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.532306    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-kube-api-access-4bzgb" (OuterVolumeSpecName: "kube-api-access-4bzgb") pod "3aa66a09-17c5-4e65-83b5-88c1987dd4e7" (UID: "3aa66a09-17c5-4e65-83b5-88c1987dd4e7"). InnerVolumeSpecName "kube-api-access-4bzgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:33:20.532494 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.532450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-dshm" (OuterVolumeSpecName: "dshm") pod "3aa66a09-17c5-4e65-83b5-88c1987dd4e7" (UID: "3aa66a09-17c5-4e65-83b5-88c1987dd4e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:20.532547 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.532524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3aa66a09-17c5-4e65-83b5-88c1987dd4e7" (UID: "3aa66a09-17c5-4e65-83b5-88c1987dd4e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:33:20.631351 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.631328    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:20.631467 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.631352    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bzgb\" (UniqueName: \"kubernetes.io/projected/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-kube-api-access-4bzgb\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:20.631467 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.631364    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa66a09-17c5-4e65-83b5-88c1987dd4e7-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:33:20.789426 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.789334    2563 generic.go:356] "Generic (PLEG): container finished" podID="3aa66a09-17c5-4e65-83b5-88c1987dd4e7" containerID="ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749" exitCode=0
Sep 11 18:33:20.789426 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.789367    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"
Sep 11 18:33:20.789426 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.789408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65" event={"ID":"3aa66a09-17c5-4e65-83b5-88c1987dd4e7","Type":"ContainerDied","Data":"ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749"}
Sep 11 18:33:20.789598 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.789443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65" event={"ID":"3aa66a09-17c5-4e65-83b5-88c1987dd4e7","Type":"ContainerDied","Data":"7a128eaccfb6ee5622d21913eb1ed111ee2f3eeaf8b31060c63805b8ff835856"}
Sep 11 18:33:20.789598 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.789461    2563 scope.go:117] "RemoveContainer" containerID="ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749"
Sep 11 18:33:20.802858 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.802837    2563 scope.go:117] "RemoveContainer" containerID="ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749"
Sep 11 18:33:20.803162 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:33:20.803135    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749\": container with ID starting with ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749 not found: ID does not exist" containerID="ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749"
Sep 11 18:33:20.803249 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.803167    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749"} err="failed to get container status \"ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749\": rpc error: code = NotFound desc = could not find container \"ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749\": container with ID starting with ab76b1c51b8a11f714ab41044f86d3d560a55c71b939365832bdf54e52262749 not found: ID does not exist"
Sep 11 18:33:20.812514 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.812490    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"]
Sep 11 18:33:20.816834 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:20.816813    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-71b1c2/autores-model-71b1c2-kserve-678d79c995-cdw65"]
Sep 11 18:33:21.112826 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:21.112764    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aa66a09-17c5-4e65-83b5-88c1987dd4e7" path="/var/lib/kubelet/pods/3aa66a09-17c5-4e65-83b5-88c1987dd4e7/volumes"
Sep 11 18:33:22.148716 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.148642    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"]
Sep 11 18:33:22.149039 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.148999    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffcf6b9-3369-4fd9-a942-82cf812a10f8" containerName="payload-processing"
Sep 11 18:33:22.149039 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.149011    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffcf6b9-3369-4fd9-a942-82cf812a10f8" containerName="payload-processing"
Sep 11 18:33:22.149039 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.149038    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa66a09-17c5-4e65-83b5-88c1987dd4e7" containerName="main"
Sep 11 18:33:22.149139 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.149043    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa66a09-17c5-4e65-83b5-88c1987dd4e7" containerName="main"
Sep 11 18:33:22.149139 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.149052    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5de2750-f0e7-42da-b284-3ff9bc1576ce" containerName="maas-api"
Sep 11 18:33:22.149139 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.149058    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5de2750-f0e7-42da-b284-3ff9bc1576ce" containerName="maas-api"
Sep 11 18:33:22.149139 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.149111    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffcf6b9-3369-4fd9-a942-82cf812a10f8" containerName="payload-processing"
Sep 11 18:33:22.149139 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.149120    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa66a09-17c5-4e65-83b5-88c1987dd4e7" containerName="main"
Sep 11 18:33:22.149139 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.149128    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5de2750-f0e7-42da-b284-3ff9bc1576ce" containerName="maas-api"
Sep 11 18:33:22.153523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.153501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.156216 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.156197    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b63ece\"/\"premium-w0-b63ece-kserve-self-signed-certs\""
Sep 11 18:33:22.163994 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.163970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"]
Sep 11 18:33:22.244929 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.244897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-dshm\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.245056 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.244931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-home\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.245056 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.245013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxnm2\" (UniqueName: \"kubernetes.io/projected/cd4919b9-6ed8-4809-a682-04b13d271ab2-kube-api-access-hxnm2\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.245135 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.245075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4919b9-6ed8-4809-a682-04b13d271ab2-tls-certs\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.245135 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.245129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-model-cache\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.245209 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.245147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-tmp-dir\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.346157 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.346130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-dshm\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.346328 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.346160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-home\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.346328 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.346209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnm2\" (UniqueName: \"kubernetes.io/projected/cd4919b9-6ed8-4809-a682-04b13d271ab2-kube-api-access-hxnm2\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.346328 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.346245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4919b9-6ed8-4809-a682-04b13d271ab2-tls-certs\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.346328 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.346297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-model-cache\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.346328 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.346321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-tmp-dir\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.346661 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.346630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-home\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.346721 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.346696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-tmp-dir\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.346788 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.346765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-model-cache\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.348305 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.348273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-dshm\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.348711 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.348695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4919b9-6ed8-4809-a682-04b13d271ab2-tls-certs\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.354931 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.354910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnm2\" (UniqueName: \"kubernetes.io/projected/cd4919b9-6ed8-4809-a682-04b13d271ab2-kube-api-access-hxnm2\") pod \"premium-w0-b63ece-kserve-864b69688d-6gl9h\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") " pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.466225 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.466205    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:22.735556 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.735526    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"]
Sep 11 18:33:22.737110 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:33:22.737085    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd4919b9_6ed8_4809_a682_04b13d271ab2.slice/crio-be05ed21e9e20c298186fb79b6bd0675049a0b7f3c668091a189d7291a18e35c WatchSource:0}: Error finding container be05ed21e9e20c298186fb79b6bd0675049a0b7f3c668091a189d7291a18e35c: Status 404 returned error can't find the container with id be05ed21e9e20c298186fb79b6bd0675049a0b7f3c668091a189d7291a18e35c
Sep 11 18:33:22.801728 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.801702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h" event={"ID":"cd4919b9-6ed8-4809-a682-04b13d271ab2","Type":"ContainerStarted","Data":"b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9"}
Sep 11 18:33:22.801820 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.801740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h" event={"ID":"cd4919b9-6ed8-4809-a682-04b13d271ab2","Type":"ContainerStarted","Data":"be05ed21e9e20c298186fb79b6bd0675049a0b7f3c668091a189d7291a18e35c"}
Sep 11 18:33:22.817757 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:22.817711    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h" podStartSLOduration=0.817696581 podStartE2EDuration="817.696581ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:22.816846433 +0000 UTC m=+1058.211361669" watchObservedRunningTime="2026-09-11 18:33:22.817696581 +0000 UTC m=+1058.212211649"
Sep 11 18:33:30.351527 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:30.351491    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:33:32.466872 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:32.466836    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:32.467310 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:32.466888    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:32.479258 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:32.479234    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:32.865088 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:32.865020    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:33:32.970550 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:32.970520    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"]
Sep 11 18:33:32.974294 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:32.974277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:33:32.976814 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:32.976791    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-54fc63\""
Sep 11 18:33:32.976944 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:32.976819    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-54fc63-dockercfg-jrhlf\""
Sep 11 18:33:32.981810 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:32.981790    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"]
Sep 11 18:33:33.033157 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:33.033131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mc85\" (UniqueName: \"kubernetes.io/projected/63510b4f-57fd-4313-b0a7-b7f7f44738cb-kube-api-access-4mc85\") pod \"payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8\" (UID: \"63510b4f-57fd-4313-b0a7-b7f7f44738cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:33:33.033261 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:33.033187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63510b4f-57fd-4313-b0a7-b7f7f44738cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8\" (UID: \"63510b4f-57fd-4313-b0a7-b7f7f44738cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:33:33.134634 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:33.134575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mc85\" (UniqueName: \"kubernetes.io/projected/63510b4f-57fd-4313-b0a7-b7f7f44738cb-kube-api-access-4mc85\") pod \"payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8\" (UID: \"63510b4f-57fd-4313-b0a7-b7f7f44738cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:33:33.134634 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:33.134621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63510b4f-57fd-4313-b0a7-b7f7f44738cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8\" (UID: \"63510b4f-57fd-4313-b0a7-b7f7f44738cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:33:33.135205 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:33.135187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63510b4f-57fd-4313-b0a7-b7f7f44738cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8\" (UID: \"63510b4f-57fd-4313-b0a7-b7f7f44738cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:33:33.143174 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:33.143149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mc85\" (UniqueName: \"kubernetes.io/projected/63510b4f-57fd-4313-b0a7-b7f7f44738cb-kube-api-access-4mc85\") pod \"payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8\" (UID: \"63510b4f-57fd-4313-b0a7-b7f7f44738cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:33:33.286311 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:33.286292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:33:33.560817 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:33.560791    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"]
Sep 11 18:33:33.562484 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:33:33.562458    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63510b4f_57fd_4313_b0a7_b7f7f44738cb.slice/crio-40cedd6347a98395c3f101d3c36746655c0e23b17bd6acdb5ad00c4a1454521b WatchSource:0}: Error finding container 40cedd6347a98395c3f101d3c36746655c0e23b17bd6acdb5ad00c4a1454521b: Status 404 returned error can't find the container with id 40cedd6347a98395c3f101d3c36746655c0e23b17bd6acdb5ad00c4a1454521b
Sep 11 18:33:33.862242 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:33.862161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8" event={"ID":"63510b4f-57fd-4313-b0a7-b7f7f44738cb","Type":"ContainerStarted","Data":"1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e"}
Sep 11 18:33:33.862242 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:33.862212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8" event={"ID":"63510b4f-57fd-4313-b0a7-b7f7f44738cb","Type":"ContainerStarted","Data":"40cedd6347a98395c3f101d3c36746655c0e23b17bd6acdb5ad00c4a1454521b"}
Sep 11 18:33:33.878047 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:33.877979    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8" podStartSLOduration=1.8779646030000001 podStartE2EDuration="1.877964603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:33.877097993 +0000 UTC m=+1069.271613028" watchObservedRunningTime="2026-09-11 18:33:33.877964603 +0000 UTC m=+1069.272479625"
Sep 11 18:33:34.867522 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:34.867495    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:33:45.874818 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:45.874793    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:33:51.144950 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:33:51.144914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:34:04.249367 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:04.249334    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:34:15.543186 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.543151    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"]
Sep 11 18:34:15.546799 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.546782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.549615 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.549594    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c8b274\"/\"premium-w3-c8b274-kserve-self-signed-certs\""
Sep 11 18:34:15.559941 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.559919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"]
Sep 11 18:34:15.681233 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.681209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89933326-0e42-4955-b4fc-10fa4de028b0-tls-certs\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.681332 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.681242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xmw\" (UniqueName: \"kubernetes.io/projected/89933326-0e42-4955-b4fc-10fa4de028b0-kube-api-access-92xmw\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.681332 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.681268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-model-cache\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.681332 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.681291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-home\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.681463 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.681345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-dshm\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.681463 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.681398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-tmp-dir\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.781744 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.781720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89933326-0e42-4955-b4fc-10fa4de028b0-tls-certs\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.781835 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.781761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xmw\" (UniqueName: \"kubernetes.io/projected/89933326-0e42-4955-b4fc-10fa4de028b0-kube-api-access-92xmw\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.781835 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.781788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-model-cache\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.781908 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.781889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-home\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.781967 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.781946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-dshm\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.782090 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.782004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-tmp-dir\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.782154 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.782127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-model-cache\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.782233 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.782215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-home\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.782328 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.782311    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-tmp-dir\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.784085 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.784066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-dshm\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.784331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.784315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89933326-0e42-4955-b4fc-10fa4de028b0-tls-certs\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.790329 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.790310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xmw\" (UniqueName: \"kubernetes.io/projected/89933326-0e42-4955-b4fc-10fa4de028b0-kube-api-access-92xmw\") pod \"premium-w3-c8b274-kserve-9bcfbb789-kl5ss\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") " pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:15.855183 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:15.855121    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:16.123933 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:16.123911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"]
Sep 11 18:34:16.126118 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:34:16.126094    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89933326_0e42_4955_b4fc_10fa4de028b0.slice/crio-6a209690b1053f11e7b38b8b1fe3ce209350258114724a2c92e40f16755e4bd8 WatchSource:0}: Error finding container 6a209690b1053f11e7b38b8b1fe3ce209350258114724a2c92e40f16755e4bd8: Status 404 returned error can't find the container with id 6a209690b1053f11e7b38b8b1fe3ce209350258114724a2c92e40f16755e4bd8
Sep 11 18:34:17.079802 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:17.079767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss" event={"ID":"89933326-0e42-4955-b4fc-10fa4de028b0","Type":"ContainerStarted","Data":"6bad9b7d0cd265cfc12a689ca1c3a3dfba0decc286feab6727ac1f6c29dfc680"}
Sep 11 18:34:17.080220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:17.079810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss" event={"ID":"89933326-0e42-4955-b4fc-10fa4de028b0","Type":"ContainerStarted","Data":"6a209690b1053f11e7b38b8b1fe3ce209350258114724a2c92e40f16755e4bd8"}
Sep 11 18:34:17.100105 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:17.100064    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss" podStartSLOduration=2.100050418 podStartE2EDuration="2.100050418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:17.098847178 +0000 UTC m=+1112.493362250" watchObservedRunningTime="2026-09-11 18:34:17.100050418 +0000 UTC m=+1112.494565442"
Sep 11 18:34:23.569366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:23.569326    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:34:24.452811 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.452783    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"]
Sep 11 18:34:24.456677 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.456657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:34:24.459287 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.459264    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-489fe7a4-dockercfg-vfb5r\""
Sep 11 18:34:24.459441 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.459399    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-489fe7a4\""
Sep 11 18:34:24.469984 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.469958    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"]
Sep 11 18:34:24.482624 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.482604    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"]
Sep 11 18:34:24.482718 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.482685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:34:24.491422 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.491339    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"]
Sep 11 18:34:24.553175 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.553150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1713ae65-15f9-4f5e-a543-4dbea5b84af4-plugins-config-volume\") pod \"payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9\" (UID: \"1713ae65-15f9-4f5e-a543-4dbea5b84af4\") " pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:34:24.553283 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.553184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgqv\" (UniqueName: \"kubernetes.io/projected/1713ae65-15f9-4f5e-a543-4dbea5b84af4-kube-api-access-9zgqv\") pod \"payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9\" (UID: \"1713ae65-15f9-4f5e-a543-4dbea5b84af4\") " pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:34:24.553283 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.553217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d\" (UID: \"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:34:24.553283 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.553266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmmq2\" (UniqueName: \"kubernetes.io/projected/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-kube-api-access-lmmq2\") pod \"payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d\" (UID: \"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:34:24.653892 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.653868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1713ae65-15f9-4f5e-a543-4dbea5b84af4-plugins-config-volume\") pod \"payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9\" (UID: \"1713ae65-15f9-4f5e-a543-4dbea5b84af4\") " pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:34:24.654213 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.653904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgqv\" (UniqueName: \"kubernetes.io/projected/1713ae65-15f9-4f5e-a543-4dbea5b84af4-kube-api-access-9zgqv\") pod \"payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9\" (UID: \"1713ae65-15f9-4f5e-a543-4dbea5b84af4\") " pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:34:24.654213 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.653921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d\" (UID: \"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:34:24.654213 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.653969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmmq2\" (UniqueName: \"kubernetes.io/projected/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-kube-api-access-lmmq2\") pod \"payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d\" (UID: \"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:34:24.654606 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.654572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1713ae65-15f9-4f5e-a543-4dbea5b84af4-plugins-config-volume\") pod \"payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9\" (UID: \"1713ae65-15f9-4f5e-a543-4dbea5b84af4\") " pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:34:24.654776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.654757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d\" (UID: \"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:34:24.661915 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.661894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmmq2\" (UniqueName: \"kubernetes.io/projected/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-kube-api-access-lmmq2\") pod \"payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d\" (UID: \"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:34:24.661999 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.661926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgqv\" (UniqueName: \"kubernetes.io/projected/1713ae65-15f9-4f5e-a543-4dbea5b84af4-kube-api-access-9zgqv\") pod \"payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9\" (UID: \"1713ae65-15f9-4f5e-a543-4dbea5b84af4\") " pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:34:24.767030 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.766978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:34:24.796092 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:24.796071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:34:25.039219 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:25.039195    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"]
Sep 11 18:34:25.040643 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:34:25.040620    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f8488e_97a3_4751_bb2e_5fdb47e2b78e.slice/crio-1ec608ed561eb60144dc036136a1d446eae2db932c5b72ba09a007ac057e9d43 WatchSource:0}: Error finding container 1ec608ed561eb60144dc036136a1d446eae2db932c5b72ba09a007ac057e9d43: Status 404 returned error can't find the container with id 1ec608ed561eb60144dc036136a1d446eae2db932c5b72ba09a007ac057e9d43
Sep 11 18:34:25.081519 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:25.081498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"]
Sep 11 18:34:25.083465 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:34:25.083433    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1713ae65_15f9_4f5e_a543_4dbea5b84af4.slice/crio-42d77313058a3ded8e07cdfa78b2009fed93a74cbd9a4f9e14c87302b71bb691 WatchSource:0}: Error finding container 42d77313058a3ded8e07cdfa78b2009fed93a74cbd9a4f9e14c87302b71bb691: Status 404 returned error can't find the container with id 42d77313058a3ded8e07cdfa78b2009fed93a74cbd9a4f9e14c87302b71bb691
Sep 11 18:34:25.123842 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:25.123812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9" event={"ID":"1713ae65-15f9-4f5e-a543-4dbea5b84af4","Type":"ContainerStarted","Data":"42d77313058a3ded8e07cdfa78b2009fed93a74cbd9a4f9e14c87302b71bb691"}
Sep 11 18:34:25.125727 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:25.125691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d" event={"ID":"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e","Type":"ContainerStarted","Data":"b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411"}
Sep 11 18:34:25.125828 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:25.125773    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:34:25.125828 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:25.125803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d" event={"ID":"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e","Type":"ContainerStarted","Data":"1ec608ed561eb60144dc036136a1d446eae2db932c5b72ba09a007ac057e9d43"}
Sep 11 18:34:25.146149 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:25.146107    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d" podStartSLOduration=1.14609303 podStartE2EDuration="1.14609303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:25.143648028 +0000 UTC m=+1120.538163078" watchObservedRunningTime="2026-09-11 18:34:25.14609303 +0000 UTC m=+1120.540608098"
Sep 11 18:34:25.856153 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:25.856122    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:25.856537 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:25.856166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:25.868613 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:25.868589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:26.132648 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:26.132584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9" event={"ID":"1713ae65-15f9-4f5e-a543-4dbea5b84af4","Type":"ContainerStarted","Data":"bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d"}
Sep 11 18:34:26.132783 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:26.132740    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:34:26.143712 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:26.143687    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:34:26.151723 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:26.151683    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9" podStartSLOduration=2.1516699360000002 podStartE2EDuration="2.151669936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:26.150635958 +0000 UTC m=+1121.545150983" watchObservedRunningTime="2026-09-11 18:34:26.151669936 +0000 UTC m=+1121.546184960"
Sep 11 18:34:36.134874 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:36.134843    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:34:37.138757 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:37.138727    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:34:40.643717 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:34:40.643683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:35:07.256940 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:07.256889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:35:07.874322 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:07.874235    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"]
Sep 11 18:35:07.874573 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:07.874538    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9" podUID="1713ae65-15f9-4f5e-a543-4dbea5b84af4" containerName="payload-processing" containerID="cri-o://bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d" gracePeriod=30
Sep 11 18:35:07.883034 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:07.883007    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"]
Sep 11 18:35:07.883282 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:07.883232    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d" podUID="e8f8488e-97a3-4751-bb2e-5fdb47e2b78e" containerName="payload-pre-processing" containerID="cri-o://b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411" gracePeriod=30
Sep 11 18:35:08.173348 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.173322    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:35:08.176421 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.176405    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:35:08.327299 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.327268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-plugins-config-volume\") pod \"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e\" (UID: \"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e\") "
Sep 11 18:35:08.327756 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.327405    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmmq2\" (UniqueName: \"kubernetes.io/projected/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-kube-api-access-lmmq2\") pod \"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e\" (UID: \"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e\") "
Sep 11 18:35:08.327756 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.327449    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zgqv\" (UniqueName: \"kubernetes.io/projected/1713ae65-15f9-4f5e-a543-4dbea5b84af4-kube-api-access-9zgqv\") pod \"1713ae65-15f9-4f5e-a543-4dbea5b84af4\" (UID: \"1713ae65-15f9-4f5e-a543-4dbea5b84af4\") "
Sep 11 18:35:08.327756 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.327487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1713ae65-15f9-4f5e-a543-4dbea5b84af4-plugins-config-volume\") pod \"1713ae65-15f9-4f5e-a543-4dbea5b84af4\" (UID: \"1713ae65-15f9-4f5e-a543-4dbea5b84af4\") "
Sep 11 18:35:08.327756 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.327674    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8f8488e-97a3-4751-bb2e-5fdb47e2b78e" (UID: "e8f8488e-97a3-4751-bb2e-5fdb47e2b78e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:08.327986 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.327840    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:35:08.327986 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.327874    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1713ae65-15f9-4f5e-a543-4dbea5b84af4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1713ae65-15f9-4f5e-a543-4dbea5b84af4" (UID: "1713ae65-15f9-4f5e-a543-4dbea5b84af4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:08.329722 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.329697    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1713ae65-15f9-4f5e-a543-4dbea5b84af4-kube-api-access-9zgqv" (OuterVolumeSpecName: "kube-api-access-9zgqv") pod "1713ae65-15f9-4f5e-a543-4dbea5b84af4" (UID: "1713ae65-15f9-4f5e-a543-4dbea5b84af4"). InnerVolumeSpecName "kube-api-access-9zgqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:08.329817 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.329748    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-kube-api-access-lmmq2" (OuterVolumeSpecName: "kube-api-access-lmmq2") pod "e8f8488e-97a3-4751-bb2e-5fdb47e2b78e" (UID: "e8f8488e-97a3-4751-bb2e-5fdb47e2b78e"). InnerVolumeSpecName "kube-api-access-lmmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:08.344790 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.344767    2563 generic.go:356] "Generic (PLEG): container finished" podID="1713ae65-15f9-4f5e-a543-4dbea5b84af4" containerID="bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d" exitCode=0
Sep 11 18:35:08.344904 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.344822    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"
Sep 11 18:35:08.344904 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.344848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9" event={"ID":"1713ae65-15f9-4f5e-a543-4dbea5b84af4","Type":"ContainerDied","Data":"bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d"}
Sep 11 18:35:08.345022 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.344904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9" event={"ID":"1713ae65-15f9-4f5e-a543-4dbea5b84af4","Type":"ContainerDied","Data":"42d77313058a3ded8e07cdfa78b2009fed93a74cbd9a4f9e14c87302b71bb691"}
Sep 11 18:35:08.345022 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.344936    2563 scope.go:117] "RemoveContainer" containerID="bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d"
Sep 11 18:35:08.346471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.346426    2563 generic.go:356] "Generic (PLEG): container finished" podID="e8f8488e-97a3-4751-bb2e-5fdb47e2b78e" containerID="b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411" exitCode=0
Sep 11 18:35:08.346578 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.346521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d" event={"ID":"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e","Type":"ContainerDied","Data":"b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411"}
Sep 11 18:35:08.346578 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.346562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d" event={"ID":"e8f8488e-97a3-4751-bb2e-5fdb47e2b78e","Type":"ContainerDied","Data":"1ec608ed561eb60144dc036136a1d446eae2db932c5b72ba09a007ac057e9d43"}
Sep 11 18:35:08.346745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.346720    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"
Sep 11 18:35:08.354930 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.354913    2563 scope.go:117] "RemoveContainer" containerID="bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d"
Sep 11 18:35:08.355197 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:35:08.355179    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d\": container with ID starting with bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d not found: ID does not exist" containerID="bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d"
Sep 11 18:35:08.355262 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.355203    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d"} err="failed to get container status \"bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d\": rpc error: code = NotFound desc = could not find container \"bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d\": container with ID starting with bb017b3dcaf527da8de9f61958797a83a9bea2716f781e1e31694597fc6ce99d not found: ID does not exist"
Sep 11 18:35:08.355262 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.355219    2563 scope.go:117] "RemoveContainer" containerID="b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411"
Sep 11 18:35:08.363297 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.363280    2563 scope.go:117] "RemoveContainer" containerID="b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411"
Sep 11 18:35:08.363617 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:35:08.363593    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411\": container with ID starting with b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411 not found: ID does not exist" containerID="b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411"
Sep 11 18:35:08.363696 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.363626    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411"} err="failed to get container status \"b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411\": rpc error: code = NotFound desc = could not find container \"b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411\": container with ID starting with b3eb3c946ba7163162155baff53f82a0253bcc0e5747df34f889f51871138411 not found: ID does not exist"
Sep 11 18:35:08.371693 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.371673    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"]
Sep 11 18:35:08.375781 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.375761    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-489fe7a4-7fb6f6fc9c-4bvn9"]
Sep 11 18:35:08.386987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.386968    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"]
Sep 11 18:35:08.391188 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.391160    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-489fe7a4-645db94dc4-mxw5d"]
Sep 11 18:35:08.428944 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.428917    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1713ae65-15f9-4f5e-a543-4dbea5b84af4-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:35:08.429027 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.428947    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmmq2\" (UniqueName: \"kubernetes.io/projected/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e-kube-api-access-lmmq2\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:35:08.429027 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:08.428962    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zgqv\" (UniqueName: \"kubernetes.io/projected/1713ae65-15f9-4f5e-a543-4dbea5b84af4-kube-api-access-9zgqv\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:35:09.114342 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:09.114291    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1713ae65-15f9-4f5e-a543-4dbea5b84af4" path="/var/lib/kubelet/pods/1713ae65-15f9-4f5e-a543-4dbea5b84af4/volumes"
Sep 11 18:35:09.114802 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:09.114785    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f8488e-97a3-4751-bb2e-5fdb47e2b78e" path="/var/lib/kubelet/pods/e8f8488e-97a3-4751-bb2e-5fdb47e2b78e/volumes"
Sep 11 18:35:27.762684 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:27.762649    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"]
Sep 11 18:35:27.763140 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:27.762906    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8" podUID="63510b4f-57fd-4313-b0a7-b7f7f44738cb" containerName="payload-pre-processing" containerID="cri-o://1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e" gracePeriod=30
Sep 11 18:35:28.031719 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.031695    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:35:28.091498 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.091471    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mc85\" (UniqueName: \"kubernetes.io/projected/63510b4f-57fd-4313-b0a7-b7f7f44738cb-kube-api-access-4mc85\") pod \"63510b4f-57fd-4313-b0a7-b7f7f44738cb\" (UID: \"63510b4f-57fd-4313-b0a7-b7f7f44738cb\") "
Sep 11 18:35:28.091625 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.091595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63510b4f-57fd-4313-b0a7-b7f7f44738cb-plugins-config-volume\") pod \"63510b4f-57fd-4313-b0a7-b7f7f44738cb\" (UID: \"63510b4f-57fd-4313-b0a7-b7f7f44738cb\") "
Sep 11 18:35:28.091959 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.091932    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63510b4f-57fd-4313-b0a7-b7f7f44738cb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63510b4f-57fd-4313-b0a7-b7f7f44738cb" (UID: "63510b4f-57fd-4313-b0a7-b7f7f44738cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:28.093456 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.093433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63510b4f-57fd-4313-b0a7-b7f7f44738cb-kube-api-access-4mc85" (OuterVolumeSpecName: "kube-api-access-4mc85") pod "63510b4f-57fd-4313-b0a7-b7f7f44738cb" (UID: "63510b4f-57fd-4313-b0a7-b7f7f44738cb"). InnerVolumeSpecName "kube-api-access-4mc85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:28.192271 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.192240    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63510b4f-57fd-4313-b0a7-b7f7f44738cb-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:35:28.192271 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.192261    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mc85\" (UniqueName: \"kubernetes.io/projected/63510b4f-57fd-4313-b0a7-b7f7f44738cb-kube-api-access-4mc85\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:35:28.460299 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.460269    2563 generic.go:356] "Generic (PLEG): container finished" podID="63510b4f-57fd-4313-b0a7-b7f7f44738cb" containerID="1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e" exitCode=0
Sep 11 18:35:28.460429 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.460317    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"
Sep 11 18:35:28.460429 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.460349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8" event={"ID":"63510b4f-57fd-4313-b0a7-b7f7f44738cb","Type":"ContainerDied","Data":"1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e"}
Sep 11 18:35:28.460429 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.460420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8" event={"ID":"63510b4f-57fd-4313-b0a7-b7f7f44738cb","Type":"ContainerDied","Data":"40cedd6347a98395c3f101d3c36746655c0e23b17bd6acdb5ad00c4a1454521b"}
Sep 11 18:35:28.460549 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.460442    2563 scope.go:117] "RemoveContainer" containerID="1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e"
Sep 11 18:35:28.469257 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.469240    2563 scope.go:117] "RemoveContainer" containerID="1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e"
Sep 11 18:35:28.469557 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:35:28.469541    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e\": container with ID starting with 1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e not found: ID does not exist" containerID="1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e"
Sep 11 18:35:28.469613 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.469563    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e"} err="failed to get container status \"1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e\": rpc error: code = NotFound desc = could not find container \"1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e\": container with ID starting with 1e909be862334dd1fe3558523af21c23f635a34893417375fa85671eeebc235e not found: ID does not exist"
Sep 11 18:35:28.485130 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.485111    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"]
Sep 11 18:35:28.493579 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:28.493559    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-54fc63-5c7b85f594-67ml8"]
Sep 11 18:35:29.113546 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:29.113517    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63510b4f-57fd-4313-b0a7-b7f7f44738cb" path="/var/lib/kubelet/pods/63510b4f-57fd-4313-b0a7-b7f7f44738cb/volumes"
Sep 11 18:35:44.150141 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:44.150108    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:35:45.123150 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:45.123123    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:35:45.127355 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:45.127335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:35:45.131135 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:45.131114    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:35:45.135135 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:35:45.135120    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:36:06.867131 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:06.867102    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:36:15.834587 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.834555    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"]
Sep 11 18:36:15.835165 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.834984    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1713ae65-15f9-4f5e-a543-4dbea5b84af4" containerName="payload-processing"
Sep 11 18:36:15.835165 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.834997    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1713ae65-15f9-4f5e-a543-4dbea5b84af4" containerName="payload-processing"
Sep 11 18:36:15.835165 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.835006    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63510b4f-57fd-4313-b0a7-b7f7f44738cb" containerName="payload-pre-processing"
Sep 11 18:36:15.835165 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.835012    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="63510b4f-57fd-4313-b0a7-b7f7f44738cb" containerName="payload-pre-processing"
Sep 11 18:36:15.835165 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.835030    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f8488e-97a3-4751-bb2e-5fdb47e2b78e" containerName="payload-pre-processing"
Sep 11 18:36:15.835165 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.835035    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f8488e-97a3-4751-bb2e-5fdb47e2b78e" containerName="payload-pre-processing"
Sep 11 18:36:15.835165 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.835100    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f8488e-97a3-4751-bb2e-5fdb47e2b78e" containerName="payload-pre-processing"
Sep 11 18:36:15.835165 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.835107    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="63510b4f-57fd-4313-b0a7-b7f7f44738cb" containerName="payload-pre-processing"
Sep 11 18:36:15.835165 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.835115    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1713ae65-15f9-4f5e-a543-4dbea5b84af4" containerName="payload-processing"
Sep 11 18:36:15.838504 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.838481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:15.841592 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.841571    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-e6454f-serving-cert\""
Sep 11 18:36:15.853682 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.853661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"]
Sep 11 18:36:15.971941 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.971915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:15.972062 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.971963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk8gk\" (UniqueName: \"kubernetes.io/projected/52a16ae8-ad4a-4222-b766-f964212f32b8-kube-api-access-vk8gk\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:15.972062 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.971987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:15.972062 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:15.972053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a16ae8-ad4a-4222-b766-f964212f32b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:16.072701 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:16.072679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:16.072823 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:16.072725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk8gk\" (UniqueName: \"kubernetes.io/projected/52a16ae8-ad4a-4222-b766-f964212f32b8-kube-api-access-vk8gk\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:16.072823 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:16.072746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:16.072823 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:16.072795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a16ae8-ad4a-4222-b766-f964212f32b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:16.073026 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:36:16.072831    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-serving-cert: secret "maas-api-e2e-shared-b-w2-e6454f-serving-cert" not found
Sep 11 18:36:16.073026 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:36:16.072898    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls podName:52a16ae8-ad4a-4222-b766-f964212f32b8 nodeName:}" failed. No retries permitted until 2026-09-11 18:36:16.572881756 +0000 UTC m=+1231.967396778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls") pod "maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s" (UID: "52a16ae8-ad4a-4222-b766-f964212f32b8") : secret "maas-api-e2e-shared-b-w2-e6454f-serving-cert" not found
Sep 11 18:36:16.073527 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:16.073500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a16ae8-ad4a-4222-b766-f964212f32b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:16.075234 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:16.075218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:16.083345 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:16.083321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk8gk\" (UniqueName: \"kubernetes.io/projected/52a16ae8-ad4a-4222-b766-f964212f32b8-kube-api-access-vk8gk\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:16.577842 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:16.577807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:16.578021 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:36:16.577962    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-serving-cert: secret "maas-api-e2e-shared-b-w2-e6454f-serving-cert" not found
Sep 11 18:36:16.578084 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:36:16.578051    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls podName:52a16ae8-ad4a-4222-b766-f964212f32b8 nodeName:}" failed. No retries permitted until 2026-09-11 18:36:17.578020064 +0000 UTC m=+1232.972535075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls") pod "maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s" (UID: "52a16ae8-ad4a-4222-b766-f964212f32b8") : secret "maas-api-e2e-shared-b-w2-e6454f-serving-cert" not found
Sep 11 18:36:17.586414 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:17.586347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:17.588851 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:17.588826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:17.650236 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:17.650211    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:17.923688 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:17.923661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"]
Sep 11 18:36:17.925680 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:36:17.925652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a16ae8_ad4a_4222_b766_f964212f32b8.slice/crio-c984749ed977a2fe550b5189aa9aaf01f89b222616c1b260c3e6cbad24142263 WatchSource:0}: Error finding container c984749ed977a2fe550b5189aa9aaf01f89b222616c1b260c3e6cbad24142263: Status 404 returned error can't find the container with id c984749ed977a2fe550b5189aa9aaf01f89b222616c1b260c3e6cbad24142263
Sep 11 18:36:17.927061 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:17.927040    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:36:18.726989 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:18.726958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s" event={"ID":"52a16ae8-ad4a-4222-b766-f964212f32b8","Type":"ContainerStarted","Data":"8a5db80c0ccb7eaf8cac54154806dfcecb8f7705029e8a3ba4f170b70a99670b"}
Sep 11 18:36:18.727334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:18.727030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s" event={"ID":"52a16ae8-ad4a-4222-b766-f964212f32b8","Type":"ContainerStarted","Data":"c984749ed977a2fe550b5189aa9aaf01f89b222616c1b260c3e6cbad24142263"}
Sep 11 18:36:18.727334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:18.727112    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:18.745884 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:18.745840    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s" podStartSLOduration=3.40646807 podStartE2EDuration="3.745826802s" totalImagesPullingTime="339.358732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:18.744833113 +0000 UTC m=+1234.139348137" watchObservedRunningTime="2026-09-11 18:36:18.745826802 +0000 UTC m=+1234.140341827"
Sep 11 18:36:24.738953 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:24.738923    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:36:30.732944 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.732905    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"]
Sep 11 18:36:30.743747 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.743713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.745687 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.745661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"]
Sep 11 18:36:30.747183 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.747154    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c8b274\"/\"e2e-conflict-a-fbdeb325-kserve-self-signed-certs\""
Sep 11 18:36:30.797479 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.797454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f124157-8ffc-4e85-88cb-97e7aa023315-tls-certs\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.797585 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.797494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-tmp-dir\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.797585 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.797520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h46sq\" (UniqueName: \"kubernetes.io/projected/0f124157-8ffc-4e85-88cb-97e7aa023315-kube-api-access-h46sq\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.797585 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.797577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-home\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.797709 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.797607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-dshm\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.797709 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.797621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-model-cache\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.898256 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.898231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-tmp-dir\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.898416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.898268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h46sq\" (UniqueName: \"kubernetes.io/projected/0f124157-8ffc-4e85-88cb-97e7aa023315-kube-api-access-h46sq\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.898416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.898305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-home\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.898416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.898342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-dshm\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.898416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.898365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-model-cache\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.898675 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.898429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f124157-8ffc-4e85-88cb-97e7aa023315-tls-certs\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.898748 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.898725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-tmp-dir\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.898810 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.898754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-home\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.898879 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.898807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-model-cache\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.900514 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.900494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-dshm\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.900833 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.900818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f124157-8ffc-4e85-88cb-97e7aa023315-tls-certs\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:30.910587 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:30.910564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h46sq\" (UniqueName: \"kubernetes.io/projected/0f124157-8ffc-4e85-88cb-97e7aa023315-kube-api-access-h46sq\") pod \"e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") " pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:31.054348 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:31.054292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:31.327514 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:31.327488    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"]
Sep 11 18:36:31.329003 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:36:31.328970    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f124157_8ffc_4e85_88cb_97e7aa023315.slice/crio-ecbf618ca56642e3cf413d3a1477b88e2d9684ff0cd992d72544e05a15b0d386 WatchSource:0}: Error finding container ecbf618ca56642e3cf413d3a1477b88e2d9684ff0cd992d72544e05a15b0d386: Status 404 returned error can't find the container with id ecbf618ca56642e3cf413d3a1477b88e2d9684ff0cd992d72544e05a15b0d386
Sep 11 18:36:31.456009 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:31.455969    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:36:31.796367 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:31.796332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr" event={"ID":"0f124157-8ffc-4e85-88cb-97e7aa023315","Type":"ContainerStarted","Data":"0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac"}
Sep 11 18:36:31.796367 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:31.796393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr" event={"ID":"0f124157-8ffc-4e85-88cb-97e7aa023315","Type":"ContainerStarted","Data":"ecbf618ca56642e3cf413d3a1477b88e2d9684ff0cd992d72544e05a15b0d386"}
Sep 11 18:36:31.817184 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:31.817142    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr" podStartSLOduration=1.8171281270000001 podStartE2EDuration="1.817128127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:31.814992812 +0000 UTC m=+1247.209507837" watchObservedRunningTime="2026-09-11 18:36:31.817128127 +0000 UTC m=+1247.211643151"
Sep 11 18:36:41.055148 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:41.055069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:41.057702 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:41.055142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:41.067498 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:41.067477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:41.857175 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:41.857142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:36:53.758086 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:53.758050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:36:58.383397 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.383348    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"]
Sep 11 18:36:58.388241 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.388226    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.391973 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.391957    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b63ece\"/\"distinct-w0-b63ece-kserve-self-signed-certs\""
Sep 11 18:36:58.403445 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.403425    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"]
Sep 11 18:36:58.533178 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.533145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17ad8903-d052-46f4-8612-aa963d4e3ff0-tls-certs\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.533347 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.533185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-dshm\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.533347 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.533324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-tmp-dir\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.533451 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.533357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-home\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.533451 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.533407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6jjp\" (UniqueName: \"kubernetes.io/projected/17ad8903-d052-46f4-8612-aa963d4e3ff0-kube-api-access-d6jjp\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.533451 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.533433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-model-cache\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.634114 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.634051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6jjp\" (UniqueName: \"kubernetes.io/projected/17ad8903-d052-46f4-8612-aa963d4e3ff0-kube-api-access-d6jjp\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.634114 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.634084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-model-cache\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.634114 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.634104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17ad8903-d052-46f4-8612-aa963d4e3ff0-tls-certs\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.634328 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.634205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-dshm\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.634328 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.634297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-tmp-dir\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.634328 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.634322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-home\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.634561 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.634364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-model-cache\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.634642 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.634625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-tmp-dir\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.634706 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.634689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-home\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.636365 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.636345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-dshm\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.636607 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.636590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17ad8903-d052-46f4-8612-aa963d4e3ff0-tls-certs\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.644089 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.644069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6jjp\" (UniqueName: \"kubernetes.io/projected/17ad8903-d052-46f4-8612-aa963d4e3ff0-kube-api-access-d6jjp\") pod \"distinct-w0-b63ece-kserve-78c558d999-f84tg\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.697659 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.697631    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:36:58.975250 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:58.975227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"]
Sep 11 18:36:58.976860 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:36:58.976834    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17ad8903_d052_46f4_8612_aa963d4e3ff0.slice/crio-39621a36fa79ae130cc6f097786cee0361425962487495956cc6c54785d0e8d3 WatchSource:0}: Error finding container 39621a36fa79ae130cc6f097786cee0361425962487495956cc6c54785d0e8d3: Status 404 returned error can't find the container with id 39621a36fa79ae130cc6f097786cee0361425962487495956cc6c54785d0e8d3
Sep 11 18:36:59.942487 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:59.942446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg" event={"ID":"17ad8903-d052-46f4-8612-aa963d4e3ff0","Type":"ContainerStarted","Data":"73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77"}
Sep 11 18:36:59.942487 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:59.942496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg" event={"ID":"17ad8903-d052-46f4-8612-aa963d4e3ff0","Type":"ContainerStarted","Data":"39621a36fa79ae130cc6f097786cee0361425962487495956cc6c54785d0e8d3"}
Sep 11 18:36:59.962843 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:36:59.962802    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg" podStartSLOduration=1.962786814 podStartE2EDuration="1.962786814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:59.959999313 +0000 UTC m=+1275.354514349" watchObservedRunningTime="2026-09-11 18:36:59.962786814 +0000 UTC m=+1275.357301867"
Sep 11 18:37:00.509836 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.509806    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"]
Sep 11 18:37:00.513534 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.513512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.516047 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.516030    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8d2ddf84-serving-cert\""
Sep 11 18:37:00.525852 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.525826    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"]
Sep 11 18:37:00.652309 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.652281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqnmv\" (UniqueName: \"kubernetes.io/projected/7434fa7f-fe5d-4b36-95a8-723dc989b90b-kube-api-access-rqnmv\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.652309 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.652312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-metrics-certs\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.652514 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.652350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-maas-api-tls\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.652556 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.652523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7434fa7f-fe5d-4b36-95a8-723dc989b90b-openshift-service-ca\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.753082 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.753058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7434fa7f-fe5d-4b36-95a8-723dc989b90b-openshift-service-ca\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.753195 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.753095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnmv\" (UniqueName: \"kubernetes.io/projected/7434fa7f-fe5d-4b36-95a8-723dc989b90b-kube-api-access-rqnmv\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.753195 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.753120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-metrics-certs\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.753326 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.753286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-maas-api-tls\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.753789 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.753770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7434fa7f-fe5d-4b36-95a8-723dc989b90b-openshift-service-ca\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.755742 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.755723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-metrics-certs\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.755742 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.755731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-maas-api-tls\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.762056 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.762009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnmv\" (UniqueName: \"kubernetes.io/projected/7434fa7f-fe5d-4b36-95a8-723dc989b90b-kube-api-access-rqnmv\") pod \"maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:00.829215 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:00.829193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:01.101502 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:01.101481    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"]
Sep 11 18:37:01.103893 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:37:01.103857    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7434fa7f_fe5d_4b36_95a8_723dc989b90b.slice/crio-b7b49d7a0be7d3131b4bf37d663233bf6fbb51347069d2b9bcc84939d08c1739 WatchSource:0}: Error finding container b7b49d7a0be7d3131b4bf37d663233bf6fbb51347069d2b9bcc84939d08c1739: Status 404 returned error can't find the container with id b7b49d7a0be7d3131b4bf37d663233bf6fbb51347069d2b9bcc84939d08c1739
Sep 11 18:37:01.953940 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:01.953910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2" event={"ID":"7434fa7f-fe5d-4b36-95a8-723dc989b90b","Type":"ContainerStarted","Data":"bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7"}
Sep 11 18:37:01.954069 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:01.954037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:01.954069 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:01.954061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2" event={"ID":"7434fa7f-fe5d-4b36-95a8-723dc989b90b","Type":"ContainerStarted","Data":"b7b49d7a0be7d3131b4bf37d663233bf6fbb51347069d2b9bcc84939d08c1739"}
Sep 11 18:37:01.973032 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:01.972985    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2" podStartSLOduration=1.233758029 podStartE2EDuration="1.972971257s" totalImagesPullingTime="739.213228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:37:01.972392841 +0000 UTC m=+1277.366907876" watchObservedRunningTime="2026-09-11 18:37:01.972971257 +0000 UTC m=+1277.367486279"
Sep 11 18:37:07.964146 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:07.964119    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:37:08.698555 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:08.698514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:37:08.698754 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:08.698599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:37:08.712335 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:08.712307    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:37:09.008713 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:09.008643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:37:26.256271 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:26.256229    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:37:46.611258 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.611219    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"]
Sep 11 18:37:46.611727 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.611476    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx" podUID="55015ade-2977-4185-a02b-f05c8364305d" containerName="maas-api" containerID="cri-o://cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3" gracePeriod=30
Sep 11 18:37:46.739080 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.739016    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"]
Sep 11 18:37:46.739306 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.739249    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk" podUID="54590d25-3d7a-4623-bbcf-9921dd4209ba" containerName="payload-pre-processing" containerID="cri-o://a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217" gracePeriod=30
Sep 11 18:37:46.875353 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.875298    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:37:46.927935 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.927911    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk9xp\" (UniqueName: \"kubernetes.io/projected/55015ade-2977-4185-a02b-f05c8364305d-kube-api-access-hk9xp\") pod \"55015ade-2977-4185-a02b-f05c8364305d\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") "
Sep 11 18:37:46.928061 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.927960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55015ade-2977-4185-a02b-f05c8364305d-openshift-service-ca\") pod \"55015ade-2977-4185-a02b-f05c8364305d\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") "
Sep 11 18:37:46.928061 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.927978    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-metrics-certs\") pod \"55015ade-2977-4185-a02b-f05c8364305d\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") "
Sep 11 18:37:46.928061 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.928033    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-maas-api-tls\") pod \"55015ade-2977-4185-a02b-f05c8364305d\" (UID: \"55015ade-2977-4185-a02b-f05c8364305d\") "
Sep 11 18:37:46.928403 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.928360    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55015ade-2977-4185-a02b-f05c8364305d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55015ade-2977-4185-a02b-f05c8364305d" (UID: "55015ade-2977-4185-a02b-f05c8364305d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:46.930395 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.930353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55015ade-2977-4185-a02b-f05c8364305d" (UID: "55015ade-2977-4185-a02b-f05c8364305d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:37:46.932196 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.932168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "55015ade-2977-4185-a02b-f05c8364305d" (UID: "55015ade-2977-4185-a02b-f05c8364305d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:37:46.932277 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.932196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55015ade-2977-4185-a02b-f05c8364305d-kube-api-access-hk9xp" (OuterVolumeSpecName: "kube-api-access-hk9xp") pod "55015ade-2977-4185-a02b-f05c8364305d" (UID: "55015ade-2977-4185-a02b-f05c8364305d"). InnerVolumeSpecName "kube-api-access-hk9xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:46.991417 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:46.991395    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:37:47.028861 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.028838    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55015ade-2977-4185-a02b-f05c8364305d-openshift-service-ca\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:37:47.028861 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.028859    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-metrics-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:37:47.028992 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.028869    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55015ade-2977-4185-a02b-f05c8364305d-maas-api-tls\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:37:47.028992 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.028877    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk9xp\" (UniqueName: \"kubernetes.io/projected/55015ade-2977-4185-a02b-f05c8364305d-kube-api-access-hk9xp\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:37:47.129629 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.129563    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d5gd\" (UniqueName: \"kubernetes.io/projected/54590d25-3d7a-4623-bbcf-9921dd4209ba-kube-api-access-9d5gd\") pod \"54590d25-3d7a-4623-bbcf-9921dd4209ba\" (UID: \"54590d25-3d7a-4623-bbcf-9921dd4209ba\") "
Sep 11 18:37:47.129733 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.129647    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54590d25-3d7a-4623-bbcf-9921dd4209ba-plugins-config-volume\") pod \"54590d25-3d7a-4623-bbcf-9921dd4209ba\" (UID: \"54590d25-3d7a-4623-bbcf-9921dd4209ba\") "
Sep 11 18:37:47.129970 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.129949    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54590d25-3d7a-4623-bbcf-9921dd4209ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54590d25-3d7a-4623-bbcf-9921dd4209ba" (UID: "54590d25-3d7a-4623-bbcf-9921dd4209ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:47.131665 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.131645    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54590d25-3d7a-4623-bbcf-9921dd4209ba-kube-api-access-9d5gd" (OuterVolumeSpecName: "kube-api-access-9d5gd") pod "54590d25-3d7a-4623-bbcf-9921dd4209ba" (UID: "54590d25-3d7a-4623-bbcf-9921dd4209ba"). InnerVolumeSpecName "kube-api-access-9d5gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:47.186339 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.186308    2563 generic.go:356] "Generic (PLEG): container finished" podID="54590d25-3d7a-4623-bbcf-9921dd4209ba" containerID="a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217" exitCode=0
Sep 11 18:37:47.186493 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.186337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk" event={"ID":"54590d25-3d7a-4623-bbcf-9921dd4209ba","Type":"ContainerDied","Data":"a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217"}
Sep 11 18:37:47.186493 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.186395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk" event={"ID":"54590d25-3d7a-4623-bbcf-9921dd4209ba","Type":"ContainerDied","Data":"be782b7b95eaecaffadfed83189513292a332151d3b1da631655cc6475b6638a"}
Sep 11 18:37:47.186493 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.186416    2563 scope.go:117] "RemoveContainer" containerID="a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217"
Sep 11 18:37:47.186493 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.186348    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"
Sep 11 18:37:47.187798 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.187772    2563 generic.go:356] "Generic (PLEG): container finished" podID="55015ade-2977-4185-a02b-f05c8364305d" containerID="cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3" exitCode=0
Sep 11 18:37:47.187872 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.187846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx" event={"ID":"55015ade-2977-4185-a02b-f05c8364305d","Type":"ContainerDied","Data":"cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3"}
Sep 11 18:37:47.187931 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.187882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx" event={"ID":"55015ade-2977-4185-a02b-f05c8364305d","Type":"ContainerDied","Data":"5f3a8dc1bdefa36ec2a38282ac32d9fdbf27218a48f382353ea78d2de5cf2496"}
Sep 11 18:37:47.187931 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.187890    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"
Sep 11 18:37:47.195884 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.195870    2563 scope.go:117] "RemoveContainer" containerID="a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217"
Sep 11 18:37:47.196138 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:37:47.196120    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217\": container with ID starting with a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217 not found: ID does not exist" containerID="a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217"
Sep 11 18:37:47.196199 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.196145    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217"} err="failed to get container status \"a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217\": rpc error: code = NotFound desc = could not find container \"a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217\": container with ID starting with a3ec999cc7dae64d135946126690762e2e9f4c91254d99c9d992643637551217 not found: ID does not exist"
Sep 11 18:37:47.196199 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.196160    2563 scope.go:117] "RemoveContainer" containerID="cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3"
Sep 11 18:37:47.203952 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.203929    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"]
Sep 11 18:37:47.204159 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.204146    2563 scope.go:117] "RemoveContainer" containerID="cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3"
Sep 11 18:37:47.204513 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:37:47.204488    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3\": container with ID starting with cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3 not found: ID does not exist" containerID="cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3"
Sep 11 18:37:47.204606 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.204520    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3"} err="failed to get container status \"cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3\": rpc error: code = NotFound desc = could not find container \"cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3\": container with ID starting with cc30a82b8f9cf2b5095aee48e70b11eea73430cd42d81bb39e4ee4da818e1cb3 not found: ID does not exist"
Sep 11 18:37:47.207464 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.207444    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52b585-d6fdbc6f8-49ldx"]
Sep 11 18:37:47.216634 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.216566    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"]
Sep 11 18:37:47.218927 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.218904    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52b585-64f8d5854b-qmkrk"]
Sep 11 18:37:47.230596 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.230574    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d5gd\" (UniqueName: \"kubernetes.io/projected/54590d25-3d7a-4623-bbcf-9921dd4209ba-kube-api-access-9d5gd\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:37:47.230596 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:47.230602    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54590d25-3d7a-4623-bbcf-9921dd4209ba-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:37:49.114140 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:49.114108    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54590d25-3d7a-4623-bbcf-9921dd4209ba" path="/var/lib/kubelet/pods/54590d25-3d7a-4623-bbcf-9921dd4209ba/volumes"
Sep 11 18:37:49.114580 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:37:49.114563    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55015ade-2977-4185-a02b-f05c8364305d" path="/var/lib/kubelet/pods/55015ade-2977-4185-a02b-f05c8364305d/volumes"
Sep 11 18:38:02.153823 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:02.153789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:38:04.808017 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:04.807983    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"]
Sep 11 18:38:04.808485 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:04.808232    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr" podUID="660777a4-f485-4d3f-9fd3-20f355cbbc11" containerName="maas-api" containerID="cri-o://e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7" gracePeriod=30
Sep 11 18:38:04.861724 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:04.861698    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"]
Sep 11 18:38:04.861949 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:04.861925    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk" podUID="a2b21d25-e7d9-4022-9092-d30b648eace1" containerName="payload-processing" containerID="cri-o://5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60" gracePeriod=30
Sep 11 18:38:04.870273 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:04.870243    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"]
Sep 11 18:38:04.870527 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:04.870501    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d" podUID="6440c884-b9ad-423a-a548-371ede8cdb3c" containerName="payload-pre-processing" containerID="cri-o://ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68" gracePeriod=30
Sep 11 18:38:05.151729 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.151706    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:38:05.155240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.155221    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:38:05.161627 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.161609    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:38:05.281661 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.281636    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kxz4\" (UniqueName: \"kubernetes.io/projected/a2b21d25-e7d9-4022-9092-d30b648eace1-kube-api-access-4kxz4\") pod \"a2b21d25-e7d9-4022-9092-d30b648eace1\" (UID: \"a2b21d25-e7d9-4022-9092-d30b648eace1\") "
Sep 11 18:38:05.281825 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.281671    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6ztr\" (UniqueName: \"kubernetes.io/projected/660777a4-f485-4d3f-9fd3-20f355cbbc11-kube-api-access-v6ztr\") pod \"660777a4-f485-4d3f-9fd3-20f355cbbc11\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") "
Sep 11 18:38:05.281825 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.281729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6440c884-b9ad-423a-a548-371ede8cdb3c-plugins-config-volume\") pod \"6440c884-b9ad-423a-a548-371ede8cdb3c\" (UID: \"6440c884-b9ad-423a-a548-371ede8cdb3c\") "
Sep 11 18:38:05.281825 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.281756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b21d25-e7d9-4022-9092-d30b648eace1-plugins-config-volume\") pod \"a2b21d25-e7d9-4022-9092-d30b648eace1\" (UID: \"a2b21d25-e7d9-4022-9092-d30b648eace1\") "
Sep 11 18:38:05.281966 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.281884    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-metrics-certs\") pod \"660777a4-f485-4d3f-9fd3-20f355cbbc11\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") "
Sep 11 18:38:05.281966 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.281926    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-maas-api-tls\") pod \"660777a4-f485-4d3f-9fd3-20f355cbbc11\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") "
Sep 11 18:38:05.282073 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.281989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4zkk\" (UniqueName: \"kubernetes.io/projected/6440c884-b9ad-423a-a548-371ede8cdb3c-kube-api-access-h4zkk\") pod \"6440c884-b9ad-423a-a548-371ede8cdb3c\" (UID: \"6440c884-b9ad-423a-a548-371ede8cdb3c\") "
Sep 11 18:38:05.282073 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.282018    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660777a4-f485-4d3f-9fd3-20f355cbbc11-openshift-service-ca\") pod \"660777a4-f485-4d3f-9fd3-20f355cbbc11\" (UID: \"660777a4-f485-4d3f-9fd3-20f355cbbc11\") "
Sep 11 18:38:05.282277 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.282172    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6440c884-b9ad-423a-a548-371ede8cdb3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6440c884-b9ad-423a-a548-371ede8cdb3c" (UID: "6440c884-b9ad-423a-a548-371ede8cdb3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:38:05.282277 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.282226    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b21d25-e7d9-4022-9092-d30b648eace1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2b21d25-e7d9-4022-9092-d30b648eace1" (UID: "a2b21d25-e7d9-4022-9092-d30b648eace1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:38:05.282684 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.282554    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660777a4-f485-4d3f-9fd3-20f355cbbc11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "660777a4-f485-4d3f-9fd3-20f355cbbc11" (UID: "660777a4-f485-4d3f-9fd3-20f355cbbc11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:38:05.282754 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.282725    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660777a4-f485-4d3f-9fd3-20f355cbbc11-openshift-service-ca\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:38:05.282814 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.282759    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6440c884-b9ad-423a-a548-371ede8cdb3c-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:38:05.282814 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.282776    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b21d25-e7d9-4022-9092-d30b648eace1-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:38:05.284099 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.284067    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b21d25-e7d9-4022-9092-d30b648eace1-kube-api-access-4kxz4" (OuterVolumeSpecName: "kube-api-access-4kxz4") pod "a2b21d25-e7d9-4022-9092-d30b648eace1" (UID: "a2b21d25-e7d9-4022-9092-d30b648eace1"). InnerVolumeSpecName "kube-api-access-4kxz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:38:05.284225 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.284096    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660777a4-f485-4d3f-9fd3-20f355cbbc11-kube-api-access-v6ztr" (OuterVolumeSpecName: "kube-api-access-v6ztr") pod "660777a4-f485-4d3f-9fd3-20f355cbbc11" (UID: "660777a4-f485-4d3f-9fd3-20f355cbbc11"). InnerVolumeSpecName "kube-api-access-v6ztr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:38:05.284282 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.284244    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6440c884-b9ad-423a-a548-371ede8cdb3c-kube-api-access-h4zkk" (OuterVolumeSpecName: "kube-api-access-h4zkk") pod "6440c884-b9ad-423a-a548-371ede8cdb3c" (UID: "6440c884-b9ad-423a-a548-371ede8cdb3c"). InnerVolumeSpecName "kube-api-access-h4zkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:38:05.284441 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.284425    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "660777a4-f485-4d3f-9fd3-20f355cbbc11" (UID: "660777a4-f485-4d3f-9fd3-20f355cbbc11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:38:05.284506 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.284462    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "660777a4-f485-4d3f-9fd3-20f355cbbc11" (UID: "660777a4-f485-4d3f-9fd3-20f355cbbc11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:38:05.291615 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.291591    2563 generic.go:356] "Generic (PLEG): container finished" podID="6440c884-b9ad-423a-a548-371ede8cdb3c" containerID="ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68" exitCode=0
Sep 11 18:38:05.291727 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.291639    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"
Sep 11 18:38:05.291727 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.291674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d" event={"ID":"6440c884-b9ad-423a-a548-371ede8cdb3c","Type":"ContainerDied","Data":"ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68"}
Sep 11 18:38:05.291851 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.291738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d" event={"ID":"6440c884-b9ad-423a-a548-371ede8cdb3c","Type":"ContainerDied","Data":"9f74dc10bf0bd86ca8d65c51dd7365820c703c86a2a6c0dc5c9d9f430884639c"}
Sep 11 18:38:05.291851 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.291769    2563 scope.go:117] "RemoveContainer" containerID="ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68"
Sep 11 18:38:05.293041 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.293024    2563 generic.go:356] "Generic (PLEG): container finished" podID="660777a4-f485-4d3f-9fd3-20f355cbbc11" containerID="e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7" exitCode=0
Sep 11 18:38:05.293115 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.293053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr" event={"ID":"660777a4-f485-4d3f-9fd3-20f355cbbc11","Type":"ContainerDied","Data":"e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7"}
Sep 11 18:38:05.293115 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.293066    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"
Sep 11 18:38:05.293115 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.293100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr" event={"ID":"660777a4-f485-4d3f-9fd3-20f355cbbc11","Type":"ContainerDied","Data":"7005fee7b6b8274ec545eecb2e7ea0f5e47e1f4dae746c325841002e67fc9f8e"}
Sep 11 18:38:05.294807 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.294788    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"
Sep 11 18:38:05.294906 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.294805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk" event={"ID":"a2b21d25-e7d9-4022-9092-d30b648eace1","Type":"ContainerDied","Data":"5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60"}
Sep 11 18:38:05.294906 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.294750    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2b21d25-e7d9-4022-9092-d30b648eace1" containerID="5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60" exitCode=0
Sep 11 18:38:05.294996 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.294932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk" event={"ID":"a2b21d25-e7d9-4022-9092-d30b648eace1","Type":"ContainerDied","Data":"c711d97e8a753efa3935db514d7e32fd89b1ffd6e88bbbca88cc3a9200f71b96"}
Sep 11 18:38:05.301842 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.301826    2563 scope.go:117] "RemoveContainer" containerID="ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68"
Sep 11 18:38:05.302162 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:38:05.302126    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68\": container with ID starting with ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68 not found: ID does not exist" containerID="ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68"
Sep 11 18:38:05.302249 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.302166    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68"} err="failed to get container status \"ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68\": rpc error: code = NotFound desc = could not find container \"ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68\": container with ID starting with ccb0ae5734e7e2872ed53529896a64f7c4472d7baa327925eda947f699913f68 not found: ID does not exist"
Sep 11 18:38:05.302249 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.302181    2563 scope.go:117] "RemoveContainer" containerID="e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7"
Sep 11 18:38:05.311136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.311118    2563 scope.go:117] "RemoveContainer" containerID="e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7"
Sep 11 18:38:05.311364 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:38:05.311346    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7\": container with ID starting with e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7 not found: ID does not exist" containerID="e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7"
Sep 11 18:38:05.311422 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.311385    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7"} err="failed to get container status \"e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7\": rpc error: code = NotFound desc = could not find container \"e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7\": container with ID starting with e0ebfd7c5bd077811e62ed3c40488e9ed830fa272646c9c6f31e1f4533b4eca7 not found: ID does not exist"
Sep 11 18:38:05.311422 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.311401    2563 scope.go:117] "RemoveContainer" containerID="5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60"
Sep 11 18:38:05.319336 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.319316    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"]
Sep 11 18:38:05.319700 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.319684    2563 scope.go:117] "RemoveContainer" containerID="5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60"
Sep 11 18:38:05.319969 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:38:05.319945    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60\": container with ID starting with 5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60 not found: ID does not exist" containerID="5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60"
Sep 11 18:38:05.320037 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.319979    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60"} err="failed to get container status \"5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60\": rpc error: code = NotFound desc = could not find container \"5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60\": container with ID starting with 5fb7d81b8e5b6150bed53c1ffa3d92eb2a6c1bdbc488dd441d3d7318f9519a60 not found: ID does not exist"
Sep 11 18:38:05.322619 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.322598    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d18cc-55645d8c54-dm66d"]
Sep 11 18:38:05.333515 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.333467    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"]
Sep 11 18:38:05.337237 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.337218    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d18cc-7c7db54b9-878tr"]
Sep 11 18:38:05.347879 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.347855    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"]
Sep 11 18:38:05.353900 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.353882    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d18cc-58c9876557-6pnhk"]
Sep 11 18:38:05.383272 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.383253    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-metrics-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:38:05.383272 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.383271    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660777a4-f485-4d3f-9fd3-20f355cbbc11-maas-api-tls\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:38:05.383413 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.383280    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4zkk\" (UniqueName: \"kubernetes.io/projected/6440c884-b9ad-423a-a548-371ede8cdb3c-kube-api-access-h4zkk\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:38:05.383413 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.383290    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kxz4\" (UniqueName: \"kubernetes.io/projected/a2b21d25-e7d9-4022-9092-d30b648eace1-kube-api-access-4kxz4\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:38:05.383413 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:05.383299    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ztr\" (UniqueName: \"kubernetes.io/projected/660777a4-f485-4d3f-9fd3-20f355cbbc11-kube-api-access-v6ztr\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:38:07.113568 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:07.113535    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6440c884-b9ad-423a-a548-371ede8cdb3c" path="/var/lib/kubelet/pods/6440c884-b9ad-423a-a548-371ede8cdb3c/volumes"
Sep 11 18:38:07.114086 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:07.114065    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="660777a4-f485-4d3f-9fd3-20f355cbbc11" path="/var/lib/kubelet/pods/660777a4-f485-4d3f-9fd3-20f355cbbc11/volumes"
Sep 11 18:38:07.114498 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:07.114462    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b21d25-e7d9-4022-9092-d30b648eace1" path="/var/lib/kubelet/pods/a2b21d25-e7d9-4022-9092-d30b648eace1/volumes"
Sep 11 18:38:15.557540 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.557458    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"]
Sep 11 18:38:15.558000 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.557981    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55015ade-2977-4185-a02b-f05c8364305d" containerName="maas-api"
Sep 11 18:38:15.558075 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558003    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="55015ade-2977-4185-a02b-f05c8364305d" containerName="maas-api"
Sep 11 18:38:15.558075 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558028    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b21d25-e7d9-4022-9092-d30b648eace1" containerName="payload-processing"
Sep 11 18:38:15.558075 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558040    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b21d25-e7d9-4022-9092-d30b648eace1" containerName="payload-processing"
Sep 11 18:38:15.558220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558074    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660777a4-f485-4d3f-9fd3-20f355cbbc11" containerName="maas-api"
Sep 11 18:38:15.558220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558083    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="660777a4-f485-4d3f-9fd3-20f355cbbc11" containerName="maas-api"
Sep 11 18:38:15.558220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558099    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54590d25-3d7a-4623-bbcf-9921dd4209ba" containerName="payload-pre-processing"
Sep 11 18:38:15.558220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558107    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="54590d25-3d7a-4623-bbcf-9921dd4209ba" containerName="payload-pre-processing"
Sep 11 18:38:15.558220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558117    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6440c884-b9ad-423a-a548-371ede8cdb3c" containerName="payload-pre-processing"
Sep 11 18:38:15.558220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558125    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6440c884-b9ad-423a-a548-371ede8cdb3c" containerName="payload-pre-processing"
Sep 11 18:38:15.558220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558221    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="55015ade-2977-4185-a02b-f05c8364305d" containerName="maas-api"
Sep 11 18:38:15.558600 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558232    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6440c884-b9ad-423a-a548-371ede8cdb3c" containerName="payload-pre-processing"
Sep 11 18:38:15.558600 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558243    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="660777a4-f485-4d3f-9fd3-20f355cbbc11" containerName="maas-api"
Sep 11 18:38:15.558600 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558253    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b21d25-e7d9-4022-9092-d30b648eace1" containerName="payload-processing"
Sep 11 18:38:15.558600 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.558263    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="54590d25-3d7a-4623-bbcf-9921dd4209ba" containerName="payload-pre-processing"
Sep 11 18:38:15.563400 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.562614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.565861 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.565839    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b63ece\"/\"distinct-2-w0-b63ece-kserve-self-signed-certs\""
Sep 11 18:38:15.571410 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.571383    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"]
Sep 11 18:38:15.668270 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.668245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-model-cache\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.668418 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.668276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa27cc9b-981d-44b4-b521-3143eaad50d6-tls-certs\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.668418 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.668296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-home\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.668418 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.668406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-dshm\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.668531 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.668446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6cmg\" (UniqueName: \"kubernetes.io/projected/aa27cc9b-981d-44b4-b521-3143eaad50d6-kube-api-access-s6cmg\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.668531 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.668518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-tmp-dir\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.769824 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.769801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-tmp-dir\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.769925 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.769840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-model-cache\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.769925 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.769857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa27cc9b-981d-44b4-b521-3143eaad50d6-tls-certs\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.769925 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.769874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-home\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.769925 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.769905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-dshm\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.770115 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.769930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6cmg\" (UniqueName: \"kubernetes.io/projected/aa27cc9b-981d-44b4-b521-3143eaad50d6-kube-api-access-s6cmg\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.770235 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.770211    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-tmp-dir\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.770314 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.770232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-model-cache\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.770314 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.770275    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-home\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.772047 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.772015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-dshm\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.772352 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.772334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa27cc9b-981d-44b4-b521-3143eaad50d6-tls-certs\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.778614 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.778591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6cmg\" (UniqueName: \"kubernetes.io/projected/aa27cc9b-981d-44b4-b521-3143eaad50d6-kube-api-access-s6cmg\") pod \"distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") " pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:15.875968 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:15.875912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:16.151882 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:16.151860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"]
Sep 11 18:38:16.153959 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:38:16.153935    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa27cc9b_981d_44b4_b521_3143eaad50d6.slice/crio-a1c04d4732248a4c4fb727e1f5e7da50d3373e3f6dbe1200aeadf3647fde5b62 WatchSource:0}: Error finding container a1c04d4732248a4c4fb727e1f5e7da50d3373e3f6dbe1200aeadf3647fde5b62: Status 404 returned error can't find the container with id a1c04d4732248a4c4fb727e1f5e7da50d3373e3f6dbe1200aeadf3647fde5b62
Sep 11 18:38:16.356910 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:16.356872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc" event={"ID":"aa27cc9b-981d-44b4-b521-3143eaad50d6","Type":"ContainerStarted","Data":"3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8"}
Sep 11 18:38:16.357078 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:16.356919    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc" event={"ID":"aa27cc9b-981d-44b4-b521-3143eaad50d6","Type":"ContainerStarted","Data":"a1c04d4732248a4c4fb727e1f5e7da50d3373e3f6dbe1200aeadf3647fde5b62"}
Sep 11 18:38:16.394693 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:16.394646    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc" podStartSLOduration=1.394630911 podStartE2EDuration="1.394630911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:16.393790113 +0000 UTC m=+1351.788305139" watchObservedRunningTime="2026-09-11 18:38:16.394630911 +0000 UTC m=+1351.789145981"
Sep 11 18:38:20.610598 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.610566    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"]
Sep 11 18:38:20.614870 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.614850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.617642 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.617622    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-ec9107-serving-cert\""
Sep 11 18:38:20.624034 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.623831    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"]
Sep 11 18:38:20.714778 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.714751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.714891 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.714782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7ztq\" (UniqueName: \"kubernetes.io/projected/464ca645-1810-49d0-a50f-1435c35644f2-kube-api-access-j7ztq\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.714891 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.714876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464ca645-1810-49d0-a50f-1435c35644f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.714987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.714938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.816175 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.816149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.816288 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.816182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7ztq\" (UniqueName: \"kubernetes.io/projected/464ca645-1810-49d0-a50f-1435c35644f2-kube-api-access-j7ztq\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.816288 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.816256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464ca645-1810-49d0-a50f-1435c35644f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.816435 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.816287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.816958 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.816935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464ca645-1810-49d0-a50f-1435c35644f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.818630 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.818610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.818717 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.818641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.824194 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.824176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7ztq\" (UniqueName: \"kubernetes.io/projected/464ca645-1810-49d0-a50f-1435c35644f2-kube-api-access-j7ztq\") pod \"maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:20.927145 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:20.927124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:21.199398 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:21.199356    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"]
Sep 11 18:38:21.200885 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:38:21.200861    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464ca645_1810_49d0_a50f_1435c35644f2.slice/crio-d1fbf183e51f78023cbb000226ea48b0ff84a4e321f987d4ad78ab762512c50e WatchSource:0}: Error finding container d1fbf183e51f78023cbb000226ea48b0ff84a4e321f987d4ad78ab762512c50e: Status 404 returned error can't find the container with id d1fbf183e51f78023cbb000226ea48b0ff84a4e321f987d4ad78ab762512c50e
Sep 11 18:38:21.383143 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:21.383100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc" event={"ID":"464ca645-1810-49d0-a50f-1435c35644f2","Type":"ContainerStarted","Data":"d1fbf183e51f78023cbb000226ea48b0ff84a4e321f987d4ad78ab762512c50e"}
Sep 11 18:38:22.390571 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:22.390527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc" event={"ID":"464ca645-1810-49d0-a50f-1435c35644f2","Type":"ContainerStarted","Data":"642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee"}
Sep 11 18:38:22.390920 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:22.390606    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:22.410035 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:22.409987    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc" podStartSLOduration=2.0651423859999998 podStartE2EDuration="2.409971096s" totalImagesPullingTime="344.82871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:22.408458164 +0000 UTC m=+1357.802973215" watchObservedRunningTime="2026-09-11 18:38:22.409971096 +0000 UTC m=+1357.804486118"
Sep 11 18:38:25.876445 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:25.876410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:25.876445 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:25.876452    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:25.888281 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:25.888258    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:26.421007 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:26.420983    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:38:28.401225 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:28.401195    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:38:40.789306 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:40.789259    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:38:56.294925 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.294888    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"]
Sep 11 18:38:56.298716 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.298701    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:38:56.301168 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.301145    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-d6c4cc\""
Sep 11 18:38:56.301425 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.301412    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-d6c4cc-dockercfg-rtshz\""
Sep 11 18:38:56.308437 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.308415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"]
Sep 11 18:38:56.378027 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.378004    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"]
Sep 11 18:38:56.381856 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.381837    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:38:56.391929 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.391906    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"]
Sep 11 18:38:56.424826 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.424803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92692b23-1a8a-4890-8af7-e9c8da0e17b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5\" (UID: \"92692b23-1a8a-4890-8af7-e9c8da0e17b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:38:56.424931 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.424913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrk2p\" (UniqueName: \"kubernetes.io/projected/92692b23-1a8a-4890-8af7-e9c8da0e17b5-kube-api-access-xrk2p\") pod \"payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5\" (UID: \"92692b23-1a8a-4890-8af7-e9c8da0e17b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:38:56.525770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.525732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrk2p\" (UniqueName: \"kubernetes.io/projected/92692b23-1a8a-4890-8af7-e9c8da0e17b5-kube-api-access-xrk2p\") pod \"payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5\" (UID: \"92692b23-1a8a-4890-8af7-e9c8da0e17b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:38:56.525864 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.525781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsx62\" (UniqueName: \"kubernetes.io/projected/8cce9724-afd3-4690-966a-1fd6f2cd0251-kube-api-access-wsx62\") pod \"payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz\" (UID: \"8cce9724-afd3-4690-966a-1fd6f2cd0251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:38:56.525864 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.525834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92692b23-1a8a-4890-8af7-e9c8da0e17b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5\" (UID: \"92692b23-1a8a-4890-8af7-e9c8da0e17b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:38:56.525962 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.525944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cce9724-afd3-4690-966a-1fd6f2cd0251-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz\" (UID: \"8cce9724-afd3-4690-966a-1fd6f2cd0251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:38:56.526523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.526505    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92692b23-1a8a-4890-8af7-e9c8da0e17b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5\" (UID: \"92692b23-1a8a-4890-8af7-e9c8da0e17b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:38:56.534152 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.534133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrk2p\" (UniqueName: \"kubernetes.io/projected/92692b23-1a8a-4890-8af7-e9c8da0e17b5-kube-api-access-xrk2p\") pod \"payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5\" (UID: \"92692b23-1a8a-4890-8af7-e9c8da0e17b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:38:56.609595 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.609540    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:38:56.627027 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.626997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsx62\" (UniqueName: \"kubernetes.io/projected/8cce9724-afd3-4690-966a-1fd6f2cd0251-kube-api-access-wsx62\") pod \"payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz\" (UID: \"8cce9724-afd3-4690-966a-1fd6f2cd0251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:38:56.627153 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.627139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cce9724-afd3-4690-966a-1fd6f2cd0251-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz\" (UID: \"8cce9724-afd3-4690-966a-1fd6f2cd0251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:38:56.627777 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.627760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cce9724-afd3-4690-966a-1fd6f2cd0251-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz\" (UID: \"8cce9724-afd3-4690-966a-1fd6f2cd0251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:38:56.646119 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.646095    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsx62\" (UniqueName: \"kubernetes.io/projected/8cce9724-afd3-4690-966a-1fd6f2cd0251-kube-api-access-wsx62\") pod \"payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz\" (UID: \"8cce9724-afd3-4690-966a-1fd6f2cd0251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:38:56.692042 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.692014    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:38:56.885709 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.885683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"]
Sep 11 18:38:56.887315 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:38:56.887291    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92692b23_1a8a_4890_8af7_e9c8da0e17b5.slice/crio-ec355e8c6a80989ea9b2b6d877e23eda4f0311d4386d2400f5918043500b2e5e WatchSource:0}: Error finding container ec355e8c6a80989ea9b2b6d877e23eda4f0311d4386d2400f5918043500b2e5e: Status 404 returned error can't find the container with id ec355e8c6a80989ea9b2b6d877e23eda4f0311d4386d2400f5918043500b2e5e
Sep 11 18:38:56.971290 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:56.971262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"]
Sep 11 18:38:56.972341 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:38:56.972317    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cce9724_afd3_4690_966a_1fd6f2cd0251.slice/crio-729fbb775b18146ad27912485a22ebb9c5cea42885e62049f4ac8cc79c0e46a0 WatchSource:0}: Error finding container 729fbb775b18146ad27912485a22ebb9c5cea42885e62049f4ac8cc79c0e46a0: Status 404 returned error can't find the container with id 729fbb775b18146ad27912485a22ebb9c5cea42885e62049f4ac8cc79c0e46a0
Sep 11 18:38:57.577639 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:57.577607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz" event={"ID":"8cce9724-afd3-4690-966a-1fd6f2cd0251","Type":"ContainerStarted","Data":"7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe"}
Sep 11 18:38:57.578054 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:57.577668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz" event={"ID":"8cce9724-afd3-4690-966a-1fd6f2cd0251","Type":"ContainerStarted","Data":"729fbb775b18146ad27912485a22ebb9c5cea42885e62049f4ac8cc79c0e46a0"}
Sep 11 18:38:57.578054 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:57.577696    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:38:57.579395 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:57.579355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5" event={"ID":"92692b23-1a8a-4890-8af7-e9c8da0e17b5","Type":"ContainerStarted","Data":"868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e"}
Sep 11 18:38:57.579496 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:57.579418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5" event={"ID":"92692b23-1a8a-4890-8af7-e9c8da0e17b5","Type":"ContainerStarted","Data":"ec355e8c6a80989ea9b2b6d877e23eda4f0311d4386d2400f5918043500b2e5e"}
Sep 11 18:38:57.579496 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:57.579447    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:38:57.599622 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:57.599580    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz" podStartSLOduration=1.5995656839999999 podStartE2EDuration="1.599565684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:57.598699656 +0000 UTC m=+1392.993214707" watchObservedRunningTime="2026-09-11 18:38:57.599565684 +0000 UTC m=+1392.994080711"
Sep 11 18:38:57.617354 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:38:57.617310    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5" podStartSLOduration=1.617297776 podStartE2EDuration="1.617297776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:57.616416706 +0000 UTC m=+1393.010931750" watchObservedRunningTime="2026-09-11 18:38:57.617297776 +0000 UTC m=+1393.011812799"
Sep 11 18:39:08.587319 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:08.587287    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:39:08.587786 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:08.587392    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:39:15.056554 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:15.056516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:39:30.049334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:30.049301    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"]
Sep 11 18:39:45.069544 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.069443    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr" podUID="0f124157-8ffc-4e85-88cb-97e7aa023315" containerName="main" containerID="cri-o://0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac" gracePeriod=45
Sep 11 18:39:45.324864 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.324807    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:39:45.444364 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.444337    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-model-cache\") pod \"0f124157-8ffc-4e85-88cb-97e7aa023315\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") "
Sep 11 18:39:45.444522 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.444404    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-home\") pod \"0f124157-8ffc-4e85-88cb-97e7aa023315\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") "
Sep 11 18:39:45.444522 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.444420    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-dshm\") pod \"0f124157-8ffc-4e85-88cb-97e7aa023315\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") "
Sep 11 18:39:45.444522 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.444451    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h46sq\" (UniqueName: \"kubernetes.io/projected/0f124157-8ffc-4e85-88cb-97e7aa023315-kube-api-access-h46sq\") pod \"0f124157-8ffc-4e85-88cb-97e7aa023315\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") "
Sep 11 18:39:45.444522 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.444496    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f124157-8ffc-4e85-88cb-97e7aa023315-tls-certs\") pod \"0f124157-8ffc-4e85-88cb-97e7aa023315\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") "
Sep 11 18:39:45.444699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.444552    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-tmp-dir\") pod \"0f124157-8ffc-4e85-88cb-97e7aa023315\" (UID: \"0f124157-8ffc-4e85-88cb-97e7aa023315\") "
Sep 11 18:39:45.444699 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.444563    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f124157-8ffc-4e85-88cb-97e7aa023315" (UID: "0f124157-8ffc-4e85-88cb-97e7aa023315"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:39:45.444803 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.444698    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-home" (OuterVolumeSpecName: "home") pod "0f124157-8ffc-4e85-88cb-97e7aa023315" (UID: "0f124157-8ffc-4e85-88cb-97e7aa023315"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:39:45.444882 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.444852    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f124157-8ffc-4e85-88cb-97e7aa023315" (UID: "0f124157-8ffc-4e85-88cb-97e7aa023315"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:39:45.445055 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.444922    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:39:45.445055 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.444937    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:39:45.446714 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.446689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f124157-8ffc-4e85-88cb-97e7aa023315-kube-api-access-h46sq" (OuterVolumeSpecName: "kube-api-access-h46sq") pod "0f124157-8ffc-4e85-88cb-97e7aa023315" (UID: "0f124157-8ffc-4e85-88cb-97e7aa023315"). InnerVolumeSpecName "kube-api-access-h46sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:39:45.446809 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.446724    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-dshm" (OuterVolumeSpecName: "dshm") pod "0f124157-8ffc-4e85-88cb-97e7aa023315" (UID: "0f124157-8ffc-4e85-88cb-97e7aa023315"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:39:45.446809 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.446732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f124157-8ffc-4e85-88cb-97e7aa023315-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f124157-8ffc-4e85-88cb-97e7aa023315" (UID: "0f124157-8ffc-4e85-88cb-97e7aa023315"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:39:45.545773 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.545750    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:39:45.545773 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.545771    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h46sq\" (UniqueName: \"kubernetes.io/projected/0f124157-8ffc-4e85-88cb-97e7aa023315-kube-api-access-h46sq\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:39:45.545910 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.545780    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f124157-8ffc-4e85-88cb-97e7aa023315-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:39:45.545910 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.545789    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f124157-8ffc-4e85-88cb-97e7aa023315-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:39:45.839773 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.839744    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f124157-8ffc-4e85-88cb-97e7aa023315" containerID="0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac" exitCode=0
Sep 11 18:39:45.839902 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.839784    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"
Sep 11 18:39:45.839902 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.839829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr" event={"ID":"0f124157-8ffc-4e85-88cb-97e7aa023315","Type":"ContainerDied","Data":"0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac"}
Sep 11 18:39:45.839902 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.839868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr" event={"ID":"0f124157-8ffc-4e85-88cb-97e7aa023315","Type":"ContainerDied","Data":"ecbf618ca56642e3cf413d3a1477b88e2d9684ff0cd992d72544e05a15b0d386"}
Sep 11 18:39:45.839902 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.839888    2563 scope.go:117] "RemoveContainer" containerID="0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac"
Sep 11 18:39:45.849735 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.849718    2563 scope.go:117] "RemoveContainer" containerID="0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac"
Sep 11 18:39:45.849991 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:39:45.849974    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac\": container with ID starting with 0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac not found: ID does not exist" containerID="0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac"
Sep 11 18:39:45.850038 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.850009    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac"} err="failed to get container status \"0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac\": rpc error: code = NotFound desc = could not find container \"0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac\": container with ID starting with 0a13e6be8bec7106d4d406f0968db2da476f7862dbb44cd2c21ade9c437aeaac not found: ID does not exist"
Sep 11 18:39:45.867065 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.867037    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"]
Sep 11 18:39:45.868882 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:45.868861    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/e2e-conflict-a-fbdeb325-kserve-57c47fb7c4-kz5jr"]
Sep 11 18:39:47.113719 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:47.113680    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f124157-8ffc-4e85-88cb-97e7aa023315" path="/var/lib/kubelet/pods/0f124157-8ffc-4e85-88cb-97e7aa023315/volumes"
Sep 11 18:39:50.087493 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:50.087452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:39:54.776738 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:54.776703    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"]
Sep 11 18:39:54.777174 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:54.777019    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2" podUID="7434fa7f-fe5d-4b36-95a8-723dc989b90b" containerName="maas-api" containerID="cri-o://bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7" gracePeriod=30
Sep 11 18:39:55.037066 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.037014    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:39:55.126388 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.126352    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7434fa7f-fe5d-4b36-95a8-723dc989b90b-openshift-service-ca\") pod \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") "
Sep 11 18:39:55.126538 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.126435    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqnmv\" (UniqueName: \"kubernetes.io/projected/7434fa7f-fe5d-4b36-95a8-723dc989b90b-kube-api-access-rqnmv\") pod \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") "
Sep 11 18:39:55.126538 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.126469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-maas-api-tls\") pod \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") "
Sep 11 18:39:55.126659 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.126632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-metrics-certs\") pod \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\" (UID: \"7434fa7f-fe5d-4b36-95a8-723dc989b90b\") "
Sep 11 18:39:55.126797 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.126684    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7434fa7f-fe5d-4b36-95a8-723dc989b90b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7434fa7f-fe5d-4b36-95a8-723dc989b90b" (UID: "7434fa7f-fe5d-4b36-95a8-723dc989b90b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:39:55.126974 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.126957    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7434fa7f-fe5d-4b36-95a8-723dc989b90b-openshift-service-ca\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:39:55.128623 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.128595    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7434fa7f-fe5d-4b36-95a8-723dc989b90b" (UID: "7434fa7f-fe5d-4b36-95a8-723dc989b90b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:39:55.128700 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.128646    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7434fa7f-fe5d-4b36-95a8-723dc989b90b-kube-api-access-rqnmv" (OuterVolumeSpecName: "kube-api-access-rqnmv") pod "7434fa7f-fe5d-4b36-95a8-723dc989b90b" (UID: "7434fa7f-fe5d-4b36-95a8-723dc989b90b"). InnerVolumeSpecName "kube-api-access-rqnmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:39:55.128700 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.128686    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7434fa7f-fe5d-4b36-95a8-723dc989b90b" (UID: "7434fa7f-fe5d-4b36-95a8-723dc989b90b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:39:55.227591 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.227567    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-metrics-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:39:55.227681 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.227602    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqnmv\" (UniqueName: \"kubernetes.io/projected/7434fa7f-fe5d-4b36-95a8-723dc989b90b-kube-api-access-rqnmv\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:39:55.227681 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.227621    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7434fa7f-fe5d-4b36-95a8-723dc989b90b-maas-api-tls\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:39:55.900222 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.900184    2563 generic.go:356] "Generic (PLEG): container finished" podID="7434fa7f-fe5d-4b36-95a8-723dc989b90b" containerID="bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7" exitCode=0
Sep 11 18:39:55.900651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.900237    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"
Sep 11 18:39:55.900651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.900249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2" event={"ID":"7434fa7f-fe5d-4b36-95a8-723dc989b90b","Type":"ContainerDied","Data":"bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7"}
Sep 11 18:39:55.900651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.900334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2" event={"ID":"7434fa7f-fe5d-4b36-95a8-723dc989b90b","Type":"ContainerDied","Data":"b7b49d7a0be7d3131b4bf37d663233bf6fbb51347069d2b9bcc84939d08c1739"}
Sep 11 18:39:55.900651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.900365    2563 scope.go:117] "RemoveContainer" containerID="bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7"
Sep 11 18:39:55.910267 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.910247    2563 scope.go:117] "RemoveContainer" containerID="bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7"
Sep 11 18:39:55.910575 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:39:55.910556    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7\": container with ID starting with bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7 not found: ID does not exist" containerID="bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7"
Sep 11 18:39:55.910646 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.910587    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7"} err="failed to get container status \"bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7\": rpc error: code = NotFound desc = could not find container \"bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7\": container with ID starting with bdbc1648f10450f718c5e2696ade0e9b6d029987aadc8a2f646485ba3c6581e7 not found: ID does not exist"
Sep 11 18:39:55.925329 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.925306    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"]
Sep 11 18:39:55.928960 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:55.928940    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d2ddf84-679c8dd5f9-n2ch2"]
Sep 11 18:39:57.112648 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.112617    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7434fa7f-fe5d-4b36-95a8-723dc989b90b" path="/var/lib/kubelet/pods/7434fa7f-fe5d-4b36-95a8-723dc989b90b/volumes"
Sep 11 18:39:57.628175 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.628145    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"]
Sep 11 18:39:57.628526 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.628513    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7434fa7f-fe5d-4b36-95a8-723dc989b90b" containerName="maas-api"
Sep 11 18:39:57.628569 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.628528    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7434fa7f-fe5d-4b36-95a8-723dc989b90b" containerName="maas-api"
Sep 11 18:39:57.628569 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.628550    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f124157-8ffc-4e85-88cb-97e7aa023315" containerName="main"
Sep 11 18:39:57.628569 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.628556    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f124157-8ffc-4e85-88cb-97e7aa023315" containerName="main"
Sep 11 18:39:57.628666 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.628622    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f124157-8ffc-4e85-88cb-97e7aa023315" containerName="main"
Sep 11 18:39:57.628666 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.628632    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7434fa7f-fe5d-4b36-95a8-723dc989b90b" containerName="maas-api"
Sep 11 18:39:57.631530 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.631514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.634059 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.634040    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c8b274\"/\"distinct-w3-c8b274-kserve-self-signed-certs\""
Sep 11 18:39:57.643990 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.643965    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"]
Sep 11 18:39:57.748961 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.748938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-home\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.749064 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.748967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxkpk\" (UniqueName: \"kubernetes.io/projected/ec596a1f-0f31-4f47-b039-fb46fbed08c5-kube-api-access-sxkpk\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.749064 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.748991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-dshm\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.749064 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.749010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tmp-dir\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.749166 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.749107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-model-cache\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.749166 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.749145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tls-certs\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.849646 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.849625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tls-certs\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.849750 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.849689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-home\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.849750 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.849721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxkpk\" (UniqueName: \"kubernetes.io/projected/ec596a1f-0f31-4f47-b039-fb46fbed08c5-kube-api-access-sxkpk\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.849873 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.849756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-dshm\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.849873 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.849782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tmp-dir\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.849873 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.849851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-model-cache\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.850125 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.850095    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-home\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.850227 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.850171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tmp-dir\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.850227 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.850221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-model-cache\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.851885 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.851864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-dshm\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.852126 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.852110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tls-certs\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.857758 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.857733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxkpk\" (UniqueName: \"kubernetes.io/projected/ec596a1f-0f31-4f47-b039-fb46fbed08c5-kube-api-access-sxkpk\") pod \"distinct-w3-c8b274-kserve-58dbc55f48-tzx6g\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:57.940177 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:57.940153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:39:58.212085 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:58.212062    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"]
Sep 11 18:39:58.213959 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:39:58.213925    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec596a1f_0f31_4f47_b039_fb46fbed08c5.slice/crio-4b0b8e1e13960c15057716fa6ca19a5dd5eff5626cd5107c581edee66af74790 WatchSource:0}: Error finding container 4b0b8e1e13960c15057716fa6ca19a5dd5eff5626cd5107c581edee66af74790: Status 404 returned error can't find the container with id 4b0b8e1e13960c15057716fa6ca19a5dd5eff5626cd5107c581edee66af74790
Sep 11 18:39:58.919326 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:58.919295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g" event={"ID":"ec596a1f-0f31-4f47-b039-fb46fbed08c5","Type":"ContainerStarted","Data":"8ea79d0dad99522b3e48a35b33542bc760c15242208328030b09c1ffc6322cac"}
Sep 11 18:39:58.919490 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:58.919339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g" event={"ID":"ec596a1f-0f31-4f47-b039-fb46fbed08c5","Type":"ContainerStarted","Data":"4b0b8e1e13960c15057716fa6ca19a5dd5eff5626cd5107c581edee66af74790"}
Sep 11 18:39:58.937216 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:39:58.937166    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g" podStartSLOduration=1.937152445 podStartE2EDuration="1.937152445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:39:58.935553084 +0000 UTC m=+1454.330068109" watchObservedRunningTime="2026-09-11 18:39:58.937152445 +0000 UTC m=+1454.331667469"
Sep 11 18:40:07.940996 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:07.940967    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:40:07.941459 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:07.941014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:40:07.953388 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:07.953351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:40:07.977694 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:07.977666    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:40:25.553828 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:25.553790    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:40:45.155010 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:45.154980    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:40:45.160689 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:45.160665    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:40:45.163045 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:45.163024    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:40:45.168368 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:45.168351    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:40:49.432348 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.432315    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"]
Sep 11 18:40:49.436094 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.436076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.438593 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.438574    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c8b274\"/\"distinct-2-w3-c8b274-kserve-self-signed-certs\""
Sep 11 18:40:49.444945 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.444920    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"]
Sep 11 18:40:49.592016 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.591993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-model-cache\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.592121 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.592033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tmp-dir\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.592121 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.592055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn95k\" (UniqueName: \"kubernetes.io/projected/243ae99a-e8e9-4017-91e3-ff3edc7439a4-kube-api-access-wn95k\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.592198 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.592129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tls-certs\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.592198 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.592158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-dshm\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.592198 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.592174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-home\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.692675 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.692621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-model-cache\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.692675 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.692659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tmp-dir\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.692835 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.692683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn95k\" (UniqueName: \"kubernetes.io/projected/243ae99a-e8e9-4017-91e3-ff3edc7439a4-kube-api-access-wn95k\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.692835 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.692712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tls-certs\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.692835 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.692729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-dshm\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.692835 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.692743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-home\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.693066 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.693043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tmp-dir\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.693129 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.693097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-model-cache\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.693129 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.693115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-home\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.694965 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.694941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-dshm\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.695146 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.695118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tls-certs\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.700424 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.700402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn95k\" (UniqueName: \"kubernetes.io/projected/243ae99a-e8e9-4017-91e3-ff3edc7439a4-kube-api-access-wn95k\") pod \"distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") " pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:49.748451 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:49.748427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:50.020450 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:50.020423    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"]
Sep 11 18:40:50.021402 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:40:50.021356    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod243ae99a_e8e9_4017_91e3_ff3edc7439a4.slice/crio-b23c8025c6eb9163de885d4a9338cff15eb92094a0a4949dd794c2d10af5dd29 WatchSource:0}: Error finding container b23c8025c6eb9163de885d4a9338cff15eb92094a0a4949dd794c2d10af5dd29: Status 404 returned error can't find the container with id b23c8025c6eb9163de885d4a9338cff15eb92094a0a4949dd794c2d10af5dd29
Sep 11 18:40:50.190120 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:50.190086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx" event={"ID":"243ae99a-e8e9-4017-91e3-ff3edc7439a4","Type":"ContainerStarted","Data":"eaeb43af684e83977014c22729574b88447e62c889d7eb15ea173262f039082a"}
Sep 11 18:40:50.190252 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:50.190132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx" event={"ID":"243ae99a-e8e9-4017-91e3-ff3edc7439a4","Type":"ContainerStarted","Data":"b23c8025c6eb9163de885d4a9338cff15eb92094a0a4949dd794c2d10af5dd29"}
Sep 11 18:40:50.205999 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:50.205957    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx" podStartSLOduration=1.205940057 podStartE2EDuration="1.205940057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:40:50.204966421 +0000 UTC m=+1505.599481464" watchObservedRunningTime="2026-09-11 18:40:50.205940057 +0000 UTC m=+1505.600455085"
Sep 11 18:40:59.748920 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:59.748891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:59.749352 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:59.748950    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:40:59.761142 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:40:59.761118    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:41:00.249471 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:41:00.249443    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:41:03.398712 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:41:03.398678    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:41:37.057222 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:41:37.057191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:42:04.899345 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:04.899308    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"]
Sep 11 18:42:04.902839 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:04.902820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:42:04.905136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:04.905120    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6c4c23\""
Sep 11 18:42:04.912815 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:04.912795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"]
Sep 11 18:42:04.944356 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:04.944336    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"]
Sep 11 18:42:04.947605 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:04.947592    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:42:04.950256 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:04.950241    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6c4c23-dockercfg-fsqd5\""
Sep 11 18:42:04.959022 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:04.958998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"]
Sep 11 18:42:05.017014 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.016995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qglzl\" (UniqueName: \"kubernetes.io/projected/42ff299d-a52b-4f50-a14d-f452fb424934-kube-api-access-qglzl\") pod \"payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl\" (UID: \"42ff299d-a52b-4f50-a14d-f452fb424934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:42:05.017103 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.017027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4swx\" (UniqueName: \"kubernetes.io/projected/734c76d8-733b-4c56-a67c-9809df547ab3-kube-api-access-d4swx\") pod \"payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x\" (UID: \"734c76d8-733b-4c56-a67c-9809df547ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:42:05.017103 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.017069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ff299d-a52b-4f50-a14d-f452fb424934-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl\" (UID: \"42ff299d-a52b-4f50-a14d-f452fb424934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:42:05.017185 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.017113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734c76d8-733b-4c56-a67c-9809df547ab3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x\" (UID: \"734c76d8-733b-4c56-a67c-9809df547ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:42:05.118039 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.118010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ff299d-a52b-4f50-a14d-f452fb424934-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl\" (UID: \"42ff299d-a52b-4f50-a14d-f452fb424934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:42:05.118178 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.118055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734c76d8-733b-4c56-a67c-9809df547ab3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x\" (UID: \"734c76d8-733b-4c56-a67c-9809df547ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:42:05.118178 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.118095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qglzl\" (UniqueName: \"kubernetes.io/projected/42ff299d-a52b-4f50-a14d-f452fb424934-kube-api-access-qglzl\") pod \"payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl\" (UID: \"42ff299d-a52b-4f50-a14d-f452fb424934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:42:05.118178 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.118118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4swx\" (UniqueName: \"kubernetes.io/projected/734c76d8-733b-4c56-a67c-9809df547ab3-kube-api-access-d4swx\") pod \"payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x\" (UID: \"734c76d8-733b-4c56-a67c-9809df547ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:42:05.118831 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.118811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ff299d-a52b-4f50-a14d-f452fb424934-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl\" (UID: \"42ff299d-a52b-4f50-a14d-f452fb424934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:42:05.118961 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.118939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734c76d8-733b-4c56-a67c-9809df547ab3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x\" (UID: \"734c76d8-733b-4c56-a67c-9809df547ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:42:05.127271 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.127247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qglzl\" (UniqueName: \"kubernetes.io/projected/42ff299d-a52b-4f50-a14d-f452fb424934-kube-api-access-qglzl\") pod \"payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl\" (UID: \"42ff299d-a52b-4f50-a14d-f452fb424934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:42:05.127358 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.127347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4swx\" (UniqueName: \"kubernetes.io/projected/734c76d8-733b-4c56-a67c-9809df547ab3-kube-api-access-d4swx\") pod \"payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x\" (UID: \"734c76d8-733b-4c56-a67c-9809df547ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:42:05.214505 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.214485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:42:05.257311 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.257282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:42:05.488897 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.488869    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"]
Sep 11 18:42:05.490321 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:42:05.490282    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ff299d_a52b_4f50_a14d_f452fb424934.slice/crio-cddbe54a774060f4321da82d9d2b276b2764048b6564c6dcdfb095d6f2218140 WatchSource:0}: Error finding container cddbe54a774060f4321da82d9d2b276b2764048b6564c6dcdfb095d6f2218140: Status 404 returned error can't find the container with id cddbe54a774060f4321da82d9d2b276b2764048b6564c6dcdfb095d6f2218140
Sep 11 18:42:05.540007 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.539984    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"]
Sep 11 18:42:05.543514 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:42:05.543487    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734c76d8_733b_4c56_a67c_9809df547ab3.slice/crio-22b168177cd6883f2f12568d147e3001c72fc341556adacaaa62785a7511ade5 WatchSource:0}: Error finding container 22b168177cd6883f2f12568d147e3001c72fc341556adacaaa62785a7511ade5: Status 404 returned error can't find the container with id 22b168177cd6883f2f12568d147e3001c72fc341556adacaaa62785a7511ade5
Sep 11 18:42:05.572136 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.572112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl" event={"ID":"42ff299d-a52b-4f50-a14d-f452fb424934","Type":"ContainerStarted","Data":"69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7"}
Sep 11 18:42:05.572253 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.572169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl" event={"ID":"42ff299d-a52b-4f50-a14d-f452fb424934","Type":"ContainerStarted","Data":"cddbe54a774060f4321da82d9d2b276b2764048b6564c6dcdfb095d6f2218140"}
Sep 11 18:42:05.572253 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.572201    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:42:05.573561 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.573542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x" event={"ID":"734c76d8-733b-4c56-a67c-9809df547ab3","Type":"ContainerStarted","Data":"22b168177cd6883f2f12568d147e3001c72fc341556adacaaa62785a7511ade5"}
Sep 11 18:42:05.592900 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:05.592857    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl" podStartSLOduration=1.592843406 podStartE2EDuration="1.592843406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:42:05.591097802 +0000 UTC m=+1580.985612831" watchObservedRunningTime="2026-09-11 18:42:05.592843406 +0000 UTC m=+1580.987358430"
Sep 11 18:42:06.581447 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:06.581413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x" event={"ID":"734c76d8-733b-4c56-a67c-9809df547ab3","Type":"ContainerStarted","Data":"b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16"}
Sep 11 18:42:06.581791 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:06.581611    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:42:06.601291 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:06.601245    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x" podStartSLOduration=2.601231587 podStartE2EDuration="2.601231587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:42:06.600162908 +0000 UTC m=+1581.994677946" watchObservedRunningTime="2026-09-11 18:42:06.601231587 +0000 UTC m=+1581.995746657"
Sep 11 18:42:12.875491 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:12.875455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:42:16.583661 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:16.583632    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:42:17.590830 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:17.590792    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:42:39.552565 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.552495    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"]
Sep 11 18:42:39.556714 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.556695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:42:39.559351 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.559321    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-999aa4\""
Sep 11 18:42:39.559496 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.559400    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-999aa4-dockercfg-glr99\""
Sep 11 18:42:39.567716 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.567677    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"]
Sep 11 18:42:39.709730 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.709693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4\" (UID: \"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f\") " pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:42:39.709878 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.709797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxmzm\" (UniqueName: \"kubernetes.io/projected/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-kube-api-access-fxmzm\") pod \"payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4\" (UID: \"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f\") " pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:42:39.810608 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.810534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxmzm\" (UniqueName: \"kubernetes.io/projected/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-kube-api-access-fxmzm\") pod \"payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4\" (UID: \"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f\") " pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:42:39.810760 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.810614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4\" (UID: \"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f\") " pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:42:39.811310 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.811289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4\" (UID: \"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f\") " pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:42:39.818621 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.818593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxmzm\" (UniqueName: \"kubernetes.io/projected/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-kube-api-access-fxmzm\") pod \"payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4\" (UID: \"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f\") " pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:42:39.869703 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:39.869681    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:42:40.146055 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:40.146028    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"]
Sep 11 18:42:40.148607 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:42:40.148577    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fa8c78_90b6_4dac_88e0_7dbc2a389d2f.slice/crio-f0ef1f5aea09f211baaff3473b8bf130583c12811613e2d59e3689a3c4313271 WatchSource:0}: Error finding container f0ef1f5aea09f211baaff3473b8bf130583c12811613e2d59e3689a3c4313271: Status 404 returned error can't find the container with id f0ef1f5aea09f211baaff3473b8bf130583c12811613e2d59e3689a3c4313271
Sep 11 18:42:40.765138 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:40.765099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4" event={"ID":"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f","Type":"ContainerStarted","Data":"b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398"}
Sep 11 18:42:40.765545 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:40.765160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4" event={"ID":"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f","Type":"ContainerStarted","Data":"f0ef1f5aea09f211baaff3473b8bf130583c12811613e2d59e3689a3c4313271"}
Sep 11 18:42:40.765545 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:40.765210    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:42:40.784133 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:40.784087    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4" podStartSLOduration=1.784071966 podStartE2EDuration="1.784071966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:42:40.782129467 +0000 UTC m=+1616.176644503" watchObservedRunningTime="2026-09-11 18:42:40.784071966 +0000 UTC m=+1616.178587007"
Sep 11 18:42:51.772428 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:51.772397    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:42:52.252844 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:42:52.252807    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:43:05.283786 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:05.283757    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"]
Sep 11 18:43:05.290596 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:05.290568    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"]
Sep 11 18:43:05.313696 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:05.313672    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"]
Sep 11 18:43:05.323530 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:05.323507    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"]
Sep 11 18:43:20.304117 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.304055    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc" podUID="aa27cc9b-981d-44b4-b521-3143eaad50d6" containerName="main" containerID="cri-o://3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8" gracePeriod=45
Sep 11 18:43:20.309109 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.309084    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg" podUID="17ad8903-d052-46f4-8612-aa963d4e3ff0" containerName="main" containerID="cri-o://73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77" gracePeriod=45
Sep 11 18:43:20.331655 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.331607    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h" podUID="cd4919b9-6ed8-4809-a682-04b13d271ab2" containerName="main" containerID="cri-o://b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9" gracePeriod=45
Sep 11 18:43:20.340765 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.340735    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7" podUID="12878d5b-505b-4540-a7c4-2709ac559667" containerName="main" containerID="cri-o://55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139" gracePeriod=45
Sep 11 18:43:20.647156 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.647129    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:43:20.674123 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.674092    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:43:20.678556 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.678533    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:43:20.682535 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.682495    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:43:20.756998 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.756963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-home\") pod \"cd4919b9-6ed8-4809-a682-04b13d271ab2\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") "
Sep 11 18:43:20.757178 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757037    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-tmp-dir\") pod \"12878d5b-505b-4540-a7c4-2709ac559667\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") "
Sep 11 18:43:20.757178 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-model-cache\") pod \"cd4919b9-6ed8-4809-a682-04b13d271ab2\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") "
Sep 11 18:43:20.757178 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757088    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17ad8903-d052-46f4-8612-aa963d4e3ff0-tls-certs\") pod \"17ad8903-d052-46f4-8612-aa963d4e3ff0\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") "
Sep 11 18:43:20.757178 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757131    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxnm2\" (UniqueName: \"kubernetes.io/projected/cd4919b9-6ed8-4809-a682-04b13d271ab2-kube-api-access-hxnm2\") pod \"cd4919b9-6ed8-4809-a682-04b13d271ab2\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") "
Sep 11 18:43:20.757178 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757170    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6jjp\" (UniqueName: \"kubernetes.io/projected/17ad8903-d052-46f4-8612-aa963d4e3ff0-kube-api-access-d6jjp\") pod \"17ad8903-d052-46f4-8612-aa963d4e3ff0\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") "
Sep 11 18:43:20.757350 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa27cc9b-981d-44b4-b521-3143eaad50d6-tls-certs\") pod \"aa27cc9b-981d-44b4-b521-3143eaad50d6\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") "
Sep 11 18:43:20.757350 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757254    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-home\") pod \"aa27cc9b-981d-44b4-b521-3143eaad50d6\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") "
Sep 11 18:43:20.757350 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757282    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-home" (OuterVolumeSpecName: "home") pod "cd4919b9-6ed8-4809-a682-04b13d271ab2" (UID: "cd4919b9-6ed8-4809-a682-04b13d271ab2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.757350 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757293    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6cmg\" (UniqueName: \"kubernetes.io/projected/aa27cc9b-981d-44b4-b521-3143eaad50d6-kube-api-access-s6cmg\") pod \"aa27cc9b-981d-44b4-b521-3143eaad50d6\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") "
Sep 11 18:43:20.757350 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757327    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12878d5b-505b-4540-a7c4-2709ac559667" (UID: "12878d5b-505b-4540-a7c4-2709ac559667"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.757539 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757362    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfstx\" (UniqueName: \"kubernetes.io/projected/12878d5b-505b-4540-a7c4-2709ac559667-kube-api-access-qfstx\") pod \"12878d5b-505b-4540-a7c4-2709ac559667\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") "
Sep 11 18:43:20.757539 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757422    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4919b9-6ed8-4809-a682-04b13d271ab2-tls-certs\") pod \"cd4919b9-6ed8-4809-a682-04b13d271ab2\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") "
Sep 11 18:43:20.757539 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757456    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-model-cache\") pod \"17ad8903-d052-46f4-8612-aa963d4e3ff0\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") "
Sep 11 18:43:20.757539 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757494    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-model-cache\") pod \"12878d5b-505b-4540-a7c4-2709ac559667\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") "
Sep 11 18:43:20.757539 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757524    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12878d5b-505b-4540-a7c4-2709ac559667-tls-certs\") pod \"12878d5b-505b-4540-a7c4-2709ac559667\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") "
Sep 11 18:43:20.757777 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757548    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-tmp-dir\") pod \"cd4919b9-6ed8-4809-a682-04b13d271ab2\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") "
Sep 11 18:43:20.757777 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757578    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-home\") pod \"12878d5b-505b-4540-a7c4-2709ac559667\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") "
Sep 11 18:43:20.757777 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-home\") pod \"17ad8903-d052-46f4-8612-aa963d4e3ff0\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") "
Sep 11 18:43:20.757777 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757651    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-dshm\") pod \"aa27cc9b-981d-44b4-b521-3143eaad50d6\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") "
Sep 11 18:43:20.757777 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757717    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-tmp-dir\") pod \"aa27cc9b-981d-44b4-b521-3143eaad50d6\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") "
Sep 11 18:43:20.757777 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-dshm\") pod \"17ad8903-d052-46f4-8612-aa963d4e3ff0\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") "
Sep 11 18:43:20.757777 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757771    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-dshm\") pod \"cd4919b9-6ed8-4809-a682-04b13d271ab2\" (UID: \"cd4919b9-6ed8-4809-a682-04b13d271ab2\") "
Sep 11 18:43:20.758199 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757795    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-model-cache\") pod \"aa27cc9b-981d-44b4-b521-3143eaad50d6\" (UID: \"aa27cc9b-981d-44b4-b521-3143eaad50d6\") "
Sep 11 18:43:20.758199 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-dshm\") pod \"12878d5b-505b-4540-a7c4-2709ac559667\" (UID: \"12878d5b-505b-4540-a7c4-2709ac559667\") "
Sep 11 18:43:20.758199 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757981    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-tmp-dir\") pod \"17ad8903-d052-46f4-8612-aa963d4e3ff0\" (UID: \"17ad8903-d052-46f4-8612-aa963d4e3ff0\") "
Sep 11 18:43:20.758508 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.758486    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.758682 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.758668    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.762221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757787    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-model-cache" (OuterVolumeSpecName: "model-cache") pod "17ad8903-d052-46f4-8612-aa963d4e3ff0" (UID: "17ad8903-d052-46f4-8612-aa963d4e3ff0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.762221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.757888    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd4919b9-6ed8-4809-a682-04b13d271ab2" (UID: "cd4919b9-6ed8-4809-a682-04b13d271ab2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.762221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.758972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17ad8903-d052-46f4-8612-aa963d4e3ff0" (UID: "17ad8903-d052-46f4-8612-aa963d4e3ff0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.762221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.759146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa27cc9b-981d-44b4-b521-3143eaad50d6" (UID: "aa27cc9b-981d-44b4-b521-3143eaad50d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.762221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.760115    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa27cc9b-981d-44b4-b521-3143eaad50d6-kube-api-access-s6cmg" (OuterVolumeSpecName: "kube-api-access-s6cmg") pod "aa27cc9b-981d-44b4-b521-3143eaad50d6" (UID: "aa27cc9b-981d-44b4-b521-3143eaad50d6"). InnerVolumeSpecName "kube-api-access-s6cmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:20.762221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.760308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12878d5b-505b-4540-a7c4-2709ac559667-kube-api-access-qfstx" (OuterVolumeSpecName: "kube-api-access-qfstx") pod "12878d5b-505b-4540-a7c4-2709ac559667" (UID: "12878d5b-505b-4540-a7c4-2709ac559667"). InnerVolumeSpecName "kube-api-access-qfstx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:20.762221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.760319    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-home" (OuterVolumeSpecName: "home") pod "aa27cc9b-981d-44b4-b521-3143eaad50d6" (UID: "aa27cc9b-981d-44b4-b521-3143eaad50d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.762221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.760546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa27cc9b-981d-44b4-b521-3143eaad50d6" (UID: "aa27cc9b-981d-44b4-b521-3143eaad50d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.762221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.762036    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-dshm" (OuterVolumeSpecName: "dshm") pod "aa27cc9b-981d-44b4-b521-3143eaad50d6" (UID: "aa27cc9b-981d-44b4-b521-3143eaad50d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.762221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.761971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd4919b9-6ed8-4809-a682-04b13d271ab2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd4919b9-6ed8-4809-a682-04b13d271ab2" (UID: "cd4919b9-6ed8-4809-a682-04b13d271ab2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:43:20.762221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.762094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4919b9-6ed8-4809-a682-04b13d271ab2-kube-api-access-hxnm2" (OuterVolumeSpecName: "kube-api-access-hxnm2") pod "cd4919b9-6ed8-4809-a682-04b13d271ab2" (UID: "cd4919b9-6ed8-4809-a682-04b13d271ab2"). InnerVolumeSpecName "kube-api-access-hxnm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:20.762958 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.762255    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-home" (OuterVolumeSpecName: "home") pod "12878d5b-505b-4540-a7c4-2709ac559667" (UID: "12878d5b-505b-4540-a7c4-2709ac559667"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.762958 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.762334    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-model-cache" (OuterVolumeSpecName: "model-cache") pod "12878d5b-505b-4540-a7c4-2709ac559667" (UID: "12878d5b-505b-4540-a7c4-2709ac559667"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.762958 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.762559    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa27cc9b-981d-44b4-b521-3143eaad50d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa27cc9b-981d-44b4-b521-3143eaad50d6" (UID: "aa27cc9b-981d-44b4-b521-3143eaad50d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:43:20.762958 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.762563    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd4919b9-6ed8-4809-a682-04b13d271ab2" (UID: "cd4919b9-6ed8-4809-a682-04b13d271ab2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.762958 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.762668    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-home" (OuterVolumeSpecName: "home") pod "17ad8903-d052-46f4-8612-aa963d4e3ff0" (UID: "17ad8903-d052-46f4-8612-aa963d4e3ff0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.763356 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.763218    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ad8903-d052-46f4-8612-aa963d4e3ff0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17ad8903-d052-46f4-8612-aa963d4e3ff0" (UID: "17ad8903-d052-46f4-8612-aa963d4e3ff0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:43:20.763439 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.763407    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-dshm" (OuterVolumeSpecName: "dshm") pod "cd4919b9-6ed8-4809-a682-04b13d271ab2" (UID: "cd4919b9-6ed8-4809-a682-04b13d271ab2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.763658 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.763634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-dshm" (OuterVolumeSpecName: "dshm") pod "17ad8903-d052-46f4-8612-aa963d4e3ff0" (UID: "17ad8903-d052-46f4-8612-aa963d4e3ff0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.764572 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.764542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12878d5b-505b-4540-a7c4-2709ac559667-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12878d5b-505b-4540-a7c4-2709ac559667" (UID: "12878d5b-505b-4540-a7c4-2709ac559667"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:43:20.764659 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.764608    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ad8903-d052-46f4-8612-aa963d4e3ff0-kube-api-access-d6jjp" (OuterVolumeSpecName: "kube-api-access-d6jjp") pod "17ad8903-d052-46f4-8612-aa963d4e3ff0" (UID: "17ad8903-d052-46f4-8612-aa963d4e3ff0"). InnerVolumeSpecName "kube-api-access-d6jjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:20.765218 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.765195    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-dshm" (OuterVolumeSpecName: "dshm") pod "12878d5b-505b-4540-a7c4-2709ac559667" (UID: "12878d5b-505b-4540-a7c4-2709ac559667"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:43:20.860204 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860152    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860204 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860173    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860204 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860181    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860204 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860190    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17ad8903-d052-46f4-8612-aa963d4e3ff0-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860204 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860199    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxnm2\" (UniqueName: \"kubernetes.io/projected/cd4919b9-6ed8-4809-a682-04b13d271ab2-kube-api-access-hxnm2\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860204 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860207    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6jjp\" (UniqueName: \"kubernetes.io/projected/17ad8903-d052-46f4-8612-aa963d4e3ff0-kube-api-access-d6jjp\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860216    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa27cc9b-981d-44b4-b521-3143eaad50d6-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860224    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860231    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6cmg\" (UniqueName: \"kubernetes.io/projected/aa27cc9b-981d-44b4-b521-3143eaad50d6-kube-api-access-s6cmg\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860239    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfstx\" (UniqueName: \"kubernetes.io/projected/12878d5b-505b-4540-a7c4-2709ac559667-kube-api-access-qfstx\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860246    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4919b9-6ed8-4809-a682-04b13d271ab2-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860254    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860262    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860269    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12878d5b-505b-4540-a7c4-2709ac559667-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860276    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860283    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12878d5b-505b-4540-a7c4-2709ac559667-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860290    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860296    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860303    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860311    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17ad8903-d052-46f4-8612-aa963d4e3ff0-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860318    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4919b9-6ed8-4809-a682-04b13d271ab2-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.860468 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.860325    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa27cc9b-981d-44b4-b521-3143eaad50d6-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:20.975912 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.975889    2563 generic.go:356] "Generic (PLEG): container finished" podID="12878d5b-505b-4540-a7c4-2709ac559667" containerID="55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139" exitCode=0
Sep 11 18:43:20.976014 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.975940    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"
Sep 11 18:43:20.976014 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.975973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7" event={"ID":"12878d5b-505b-4540-a7c4-2709ac559667","Type":"ContainerDied","Data":"55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139"}
Sep 11 18:43:20.976014 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.976014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7" event={"ID":"12878d5b-505b-4540-a7c4-2709ac559667","Type":"ContainerDied","Data":"8cb50ec9cde1dfce96544490276ba55db79f80633e3a77ffe3e2eb0b409a116b"}
Sep 11 18:43:20.976141 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.976030    2563 scope.go:117] "RemoveContainer" containerID="55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139"
Sep 11 18:43:20.977443 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.977417    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd4919b9-6ed8-4809-a682-04b13d271ab2" containerID="b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9" exitCode=0
Sep 11 18:43:20.977555 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.977472    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"
Sep 11 18:43:20.977555 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.977500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h" event={"ID":"cd4919b9-6ed8-4809-a682-04b13d271ab2","Type":"ContainerDied","Data":"b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9"}
Sep 11 18:43:20.977555 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.977538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h" event={"ID":"cd4919b9-6ed8-4809-a682-04b13d271ab2","Type":"ContainerDied","Data":"be05ed21e9e20c298186fb79b6bd0675049a0b7f3c668091a189d7291a18e35c"}
Sep 11 18:43:20.979120 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.979093    2563 generic.go:356] "Generic (PLEG): container finished" podID="17ad8903-d052-46f4-8612-aa963d4e3ff0" containerID="73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77" exitCode=0
Sep 11 18:43:20.979221 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.979178    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"
Sep 11 18:43:20.979277 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.979213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg" event={"ID":"17ad8903-d052-46f4-8612-aa963d4e3ff0","Type":"ContainerDied","Data":"73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77"}
Sep 11 18:43:20.979277 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.979253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg" event={"ID":"17ad8903-d052-46f4-8612-aa963d4e3ff0","Type":"ContainerDied","Data":"39621a36fa79ae130cc6f097786cee0361425962487495956cc6c54785d0e8d3"}
Sep 11 18:43:20.981129 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.981098    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa27cc9b-981d-44b4-b521-3143eaad50d6" containerID="3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8" exitCode=0
Sep 11 18:43:20.981229 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.981149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc" event={"ID":"aa27cc9b-981d-44b4-b521-3143eaad50d6","Type":"ContainerDied","Data":"3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8"}
Sep 11 18:43:20.981229 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.981171    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"
Sep 11 18:43:20.981321 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.981176    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc" event={"ID":"aa27cc9b-981d-44b4-b521-3143eaad50d6","Type":"ContainerDied","Data":"a1c04d4732248a4c4fb727e1f5e7da50d3373e3f6dbe1200aeadf3647fde5b62"}
Sep 11 18:43:20.985829 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.985677    2563 scope.go:117] "RemoveContainer" containerID="55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139"
Sep 11 18:43:20.986026 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:43:20.985999    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139\": container with ID starting with 55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139 not found: ID does not exist" containerID="55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139"
Sep 11 18:43:20.986130 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.986034    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139"} err="failed to get container status \"55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139\": rpc error: code = NotFound desc = could not find container \"55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139\": container with ID starting with 55b048235c6f4ecb63e4047dab132c18331712a10d52d7086274c60ba0eda139 not found: ID does not exist"
Sep 11 18:43:20.986130 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.986058    2563 scope.go:117] "RemoveContainer" containerID="b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9"
Sep 11 18:43:20.996126 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.996104    2563 scope.go:117] "RemoveContainer" containerID="b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9"
Sep 11 18:43:20.996363 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:43:20.996347    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9\": container with ID starting with b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9 not found: ID does not exist" containerID="b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9"
Sep 11 18:43:20.996444 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.996368    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9"} err="failed to get container status \"b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9\": rpc error: code = NotFound desc = could not find container \"b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9\": container with ID starting with b2cfbd20da9d3705adb15e3878bc94796a700111f96c296bcc82dcf4ffa523d9 not found: ID does not exist"
Sep 11 18:43:20.996444 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:20.996405    2563 scope.go:117] "RemoveContainer" containerID="73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77"
Sep 11 18:43:21.004531 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.004512    2563 scope.go:117] "RemoveContainer" containerID="73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77"
Sep 11 18:43:21.004610 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.004592    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"]
Sep 11 18:43:21.004796 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:43:21.004776    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77\": container with ID starting with 73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77 not found: ID does not exist" containerID="73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77"
Sep 11 18:43:21.004859 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.004801    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77"} err="failed to get container status \"73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77\": rpc error: code = NotFound desc = could not find container \"73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77\": container with ID starting with 73130dc3bf183ad298d13226d49c3100617614c92b8332136353875100120a77 not found: ID does not exist"
Sep 11 18:43:21.004859 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.004816    2563 scope.go:117] "RemoveContainer" containerID="3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8"
Sep 11 18:43:21.006900 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.006879    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/simulator-w0-b63ece-kserve-77f5bc4b57-qsqb7"]
Sep 11 18:43:21.012408 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.012391    2563 scope.go:117] "RemoveContainer" containerID="3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8"
Sep 11 18:43:21.012621 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:43:21.012606    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8\": container with ID starting with 3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8 not found: ID does not exist" containerID="3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8"
Sep 11 18:43:21.012672 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.012625    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8"} err="failed to get container status \"3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8\": rpc error: code = NotFound desc = could not find container \"3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8\": container with ID starting with 3b52b4e49ec40e26fbd4e40dc639b091e726a84f39d2e047821bda9be033a4a8 not found: ID does not exist"
Sep 11 18:43:21.015969 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.015948    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"]
Sep 11 18:43:21.017735 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.017716    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-2-w0-b63ece-kserve-6f5cf57fb-q4ggc"]
Sep 11 18:43:21.027809 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.027784    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"]
Sep 11 18:43:21.033601 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.033582    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/premium-w0-b63ece-kserve-864b69688d-6gl9h"]
Sep 11 18:43:21.044068 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.044047    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"]
Sep 11 18:43:21.048096 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.048077    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b63ece/distinct-w0-b63ece-kserve-78c558d999-f84tg"]
Sep 11 18:43:21.114256 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.114204    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12878d5b-505b-4540-a7c4-2709ac559667" path="/var/lib/kubelet/pods/12878d5b-505b-4540-a7c4-2709ac559667/volumes"
Sep 11 18:43:21.114665 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.114650    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17ad8903-d052-46f4-8612-aa963d4e3ff0" path="/var/lib/kubelet/pods/17ad8903-d052-46f4-8612-aa963d4e3ff0/volumes"
Sep 11 18:43:21.115003 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.114991    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa27cc9b-981d-44b4-b521-3143eaad50d6" path="/var/lib/kubelet/pods/aa27cc9b-981d-44b4-b521-3143eaad50d6/volumes"
Sep 11 18:43:21.115616 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:21.115602    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd4919b9-6ed8-4809-a682-04b13d271ab2" path="/var/lib/kubelet/pods/cd4919b9-6ed8-4809-a682-04b13d271ab2/volumes"
Sep 11 18:43:32.556226 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.556193    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"]
Sep 11 18:43:32.556698 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.556432    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x" podUID="734c76d8-733b-4c56-a67c-9809df547ab3" containerName="payload-processing" containerID="cri-o://b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16" gracePeriod=30
Sep 11 18:43:32.565395 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.565290    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"]
Sep 11 18:43:32.565740 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.565688    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl" podUID="42ff299d-a52b-4f50-a14d-f452fb424934" containerName="payload-pre-processing" containerID="cri-o://69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7" gracePeriod=30
Sep 11 18:43:32.835503 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.835479    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:43:32.838501 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.838482    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:43:32.866010 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.865983    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734c76d8-733b-4c56-a67c-9809df547ab3-plugins-config-volume\") pod \"734c76d8-733b-4c56-a67c-9809df547ab3\" (UID: \"734c76d8-733b-4c56-a67c-9809df547ab3\") "
Sep 11 18:43:32.866103 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.866076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qglzl\" (UniqueName: \"kubernetes.io/projected/42ff299d-a52b-4f50-a14d-f452fb424934-kube-api-access-qglzl\") pod \"42ff299d-a52b-4f50-a14d-f452fb424934\" (UID: \"42ff299d-a52b-4f50-a14d-f452fb424934\") "
Sep 11 18:43:32.866143 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.866107    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4swx\" (UniqueName: \"kubernetes.io/projected/734c76d8-733b-4c56-a67c-9809df547ab3-kube-api-access-d4swx\") pod \"734c76d8-733b-4c56-a67c-9809df547ab3\" (UID: \"734c76d8-733b-4c56-a67c-9809df547ab3\") "
Sep 11 18:43:32.866143 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.866124    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ff299d-a52b-4f50-a14d-f452fb424934-plugins-config-volume\") pod \"42ff299d-a52b-4f50-a14d-f452fb424934\" (UID: \"42ff299d-a52b-4f50-a14d-f452fb424934\") "
Sep 11 18:43:32.866555 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.866498    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734c76d8-733b-4c56-a67c-9809df547ab3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "734c76d8-733b-4c56-a67c-9809df547ab3" (UID: "734c76d8-733b-4c56-a67c-9809df547ab3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:32.866634 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.866557    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ff299d-a52b-4f50-a14d-f452fb424934-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42ff299d-a52b-4f50-a14d-f452fb424934" (UID: "42ff299d-a52b-4f50-a14d-f452fb424934"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:32.868239 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.868215    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ff299d-a52b-4f50-a14d-f452fb424934-kube-api-access-qglzl" (OuterVolumeSpecName: "kube-api-access-qglzl") pod "42ff299d-a52b-4f50-a14d-f452fb424934" (UID: "42ff299d-a52b-4f50-a14d-f452fb424934"). InnerVolumeSpecName "kube-api-access-qglzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:32.868292 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.868235    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734c76d8-733b-4c56-a67c-9809df547ab3-kube-api-access-d4swx" (OuterVolumeSpecName: "kube-api-access-d4swx") pod "734c76d8-733b-4c56-a67c-9809df547ab3" (UID: "734c76d8-733b-4c56-a67c-9809df547ab3"). InnerVolumeSpecName "kube-api-access-d4swx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:32.967485 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.967454    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4swx\" (UniqueName: \"kubernetes.io/projected/734c76d8-733b-4c56-a67c-9809df547ab3-kube-api-access-d4swx\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:32.967485 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.967477    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ff299d-a52b-4f50-a14d-f452fb424934-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:32.967612 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.967491    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734c76d8-733b-4c56-a67c-9809df547ab3-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:32.967612 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:32.967503    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qglzl\" (UniqueName: \"kubernetes.io/projected/42ff299d-a52b-4f50-a14d-f452fb424934-kube-api-access-qglzl\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:33.049816 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.049788    2563 generic.go:356] "Generic (PLEG): container finished" podID="42ff299d-a52b-4f50-a14d-f452fb424934" containerID="69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7" exitCode=0
Sep 11 18:43:33.049901 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.049834    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"
Sep 11 18:43:33.049901 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.049878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl" event={"ID":"42ff299d-a52b-4f50-a14d-f452fb424934","Type":"ContainerDied","Data":"69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7"}
Sep 11 18:43:33.050014 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.049936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl" event={"ID":"42ff299d-a52b-4f50-a14d-f452fb424934","Type":"ContainerDied","Data":"cddbe54a774060f4321da82d9d2b276b2764048b6564c6dcdfb095d6f2218140"}
Sep 11 18:43:33.050014 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.049969    2563 scope.go:117] "RemoveContainer" containerID="69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7"
Sep 11 18:43:33.051413 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.051367    2563 generic.go:356] "Generic (PLEG): container finished" podID="734c76d8-733b-4c56-a67c-9809df547ab3" containerID="b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16" exitCode=0
Sep 11 18:43:33.051523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.051415    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"
Sep 11 18:43:33.051523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.051468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x" event={"ID":"734c76d8-733b-4c56-a67c-9809df547ab3","Type":"ContainerDied","Data":"b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16"}
Sep 11 18:43:33.051523 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.051511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x" event={"ID":"734c76d8-733b-4c56-a67c-9809df547ab3","Type":"ContainerDied","Data":"22b168177cd6883f2f12568d147e3001c72fc341556adacaaa62785a7511ade5"}
Sep 11 18:43:33.060892 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.060836    2563 scope.go:117] "RemoveContainer" containerID="69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7"
Sep 11 18:43:33.061143 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:43:33.061123    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7\": container with ID starting with 69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7 not found: ID does not exist" containerID="69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7"
Sep 11 18:43:33.061242 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.061148    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7"} err="failed to get container status \"69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7\": rpc error: code = NotFound desc = could not find container \"69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7\": container with ID starting with 69dc1158c8277be1705c9108ff92e4863d366d4870705826fde8f0e24d8792f7 not found: ID does not exist"
Sep 11 18:43:33.061242 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.061163    2563 scope.go:117] "RemoveContainer" containerID="b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16"
Sep 11 18:43:33.069949 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.069932    2563 scope.go:117] "RemoveContainer" containerID="b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16"
Sep 11 18:43:33.071138 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:43:33.071109    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16\": container with ID starting with b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16 not found: ID does not exist" containerID="b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16"
Sep 11 18:43:33.071209 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.071144    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16"} err="failed to get container status \"b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16\": rpc error: code = NotFound desc = could not find container \"b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16\": container with ID starting with b7596a218eab76eef5cda228a31e3d74b6d16d87918478c889aed1c38a983c16 not found: ID does not exist"
Sep 11 18:43:33.079592 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.079572    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"]
Sep 11 18:43:33.084169 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.084151    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6c4c23-66948bff5d-b29bl"]
Sep 11 18:43:33.101157 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.101134    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"]
Sep 11 18:43:33.113548 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.113526    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42ff299d-a52b-4f50-a14d-f452fb424934" path="/var/lib/kubelet/pods/42ff299d-a52b-4f50-a14d-f452fb424934/volumes"
Sep 11 18:43:33.113816 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:33.113800    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6c4c23-78cbb9945c-xt25x"]
Sep 11 18:43:35.112336 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:35.112306    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="734c76d8-733b-4c56-a67c-9809df547ab3" path="/var/lib/kubelet/pods/734c76d8-733b-4c56-a67c-9809df547ab3/volumes"
Sep 11 18:43:40.046669 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.046636    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"]
Sep 11 18:43:40.047206 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047189    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12878d5b-505b-4540-a7c4-2709ac559667" containerName="main"
Sep 11 18:43:40.047301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047210    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="12878d5b-505b-4540-a7c4-2709ac559667" containerName="main"
Sep 11 18:43:40.047301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047242    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd4919b9-6ed8-4809-a682-04b13d271ab2" containerName="main"
Sep 11 18:43:40.047301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047251    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd4919b9-6ed8-4809-a682-04b13d271ab2" containerName="main"
Sep 11 18:43:40.047301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047263    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17ad8903-d052-46f4-8612-aa963d4e3ff0" containerName="main"
Sep 11 18:43:40.047301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047272    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="17ad8903-d052-46f4-8612-aa963d4e3ff0" containerName="main"
Sep 11 18:43:40.047301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047281    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="734c76d8-733b-4c56-a67c-9809df547ab3" containerName="payload-processing"
Sep 11 18:43:40.047301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047289    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="734c76d8-733b-4c56-a67c-9809df547ab3" containerName="payload-processing"
Sep 11 18:43:40.047301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047302    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ff299d-a52b-4f50-a14d-f452fb424934" containerName="payload-pre-processing"
Sep 11 18:43:40.047745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047311    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ff299d-a52b-4f50-a14d-f452fb424934" containerName="payload-pre-processing"
Sep 11 18:43:40.047745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047335    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa27cc9b-981d-44b4-b521-3143eaad50d6" containerName="main"
Sep 11 18:43:40.047745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047344    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa27cc9b-981d-44b4-b521-3143eaad50d6" containerName="main"
Sep 11 18:43:40.047745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047453    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="12878d5b-505b-4540-a7c4-2709ac559667" containerName="main"
Sep 11 18:43:40.047745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047464    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="17ad8903-d052-46f4-8612-aa963d4e3ff0" containerName="main"
Sep 11 18:43:40.047745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047476    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ff299d-a52b-4f50-a14d-f452fb424934" containerName="payload-pre-processing"
Sep 11 18:43:40.047745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047487    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="734c76d8-733b-4c56-a67c-9809df547ab3" containerName="payload-processing"
Sep 11 18:43:40.047745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047498    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa27cc9b-981d-44b4-b521-3143eaad50d6" containerName="main"
Sep 11 18:43:40.047745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.047506    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd4919b9-6ed8-4809-a682-04b13d271ab2" containerName="main"
Sep 11 18:43:40.052208 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.052186    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"
Sep 11 18:43:40.054925 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.054907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:43:40.060516 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.060493    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"]
Sep 11 18:43:40.126591 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.126564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e061db-483e-49a7-b0fe-fb735107d59b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt\" (UID: \"23e061db-483e-49a7-b0fe-fb735107d59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"
Sep 11 18:43:40.227890 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.227858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e061db-483e-49a7-b0fe-fb735107d59b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt\" (UID: \"23e061db-483e-49a7-b0fe-fb735107d59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"
Sep 11 18:43:40.228447 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.228429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e061db-483e-49a7-b0fe-fb735107d59b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt\" (UID: \"23e061db-483e-49a7-b0fe-fb735107d59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"
Sep 11 18:43:40.363947 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.363898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"
Sep 11 18:43:40.640838 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:43:40.640810    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e061db_483e_49a7_b0fe_fb735107d59b.slice/crio-f18b15dc1d53b01e0eb276ed6e4c96330facd969155bf1a9010e39d94beab8aa WatchSource:0}: Error finding container f18b15dc1d53b01e0eb276ed6e4c96330facd969155bf1a9010e39d94beab8aa: Status 404 returned error can't find the container with id f18b15dc1d53b01e0eb276ed6e4c96330facd969155bf1a9010e39d94beab8aa
Sep 11 18:43:40.640838 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:40.640820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"]
Sep 11 18:43:41.096051 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:41.096021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt" event={"ID":"23e061db-483e-49a7-b0fe-fb735107d59b","Type":"ContainerStarted","Data":"93a6a127b990db1a5ce89514d9c8a9825691a74099d2bd24b613c697f7435799"}
Sep 11 18:43:41.096449 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:41.096070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt" event={"ID":"23e061db-483e-49a7-b0fe-fb735107d59b","Type":"ContainerStarted","Data":"f18b15dc1d53b01e0eb276ed6e4c96330facd969155bf1a9010e39d94beab8aa"}
Sep 11 18:43:41.114061 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:41.114009    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt" podStartSLOduration=1.113995167 podStartE2EDuration="1.113995167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:43:41.111605813 +0000 UTC m=+1676.506120864" watchObservedRunningTime="2026-09-11 18:43:41.113995167 +0000 UTC m=+1676.508510191"
Sep 11 18:43:42.102944 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:42.102910    2563 generic.go:356] "Generic (PLEG): container finished" podID="23e061db-483e-49a7-b0fe-fb735107d59b" containerID="93a6a127b990db1a5ce89514d9c8a9825691a74099d2bd24b613c697f7435799" exitCode=0
Sep 11 18:43:42.103308 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:42.102987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt" event={"ID":"23e061db-483e-49a7-b0fe-fb735107d59b","Type":"ContainerDied","Data":"93a6a127b990db1a5ce89514d9c8a9825691a74099d2bd24b613c697f7435799"}
Sep 11 18:43:43.246088 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:43.246065    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"
Sep 11 18:43:43.354362 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:43.354337    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e061db-483e-49a7-b0fe-fb735107d59b-maas-api-service-ca\") pod \"23e061db-483e-49a7-b0fe-fb735107d59b\" (UID: \"23e061db-483e-49a7-b0fe-fb735107d59b\") "
Sep 11 18:43:43.354714 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:43.354692    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e061db-483e-49a7-b0fe-fb735107d59b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23e061db-483e-49a7-b0fe-fb735107d59b" (UID: "23e061db-483e-49a7-b0fe-fb735107d59b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:43.455655 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:43.455632    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e061db-483e-49a7-b0fe-fb735107d59b-maas-api-service-ca\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:44.114661 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:44.114634    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"
Sep 11 18:43:44.114816 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:44.114686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt" event={"ID":"23e061db-483e-49a7-b0fe-fb735107d59b","Type":"ContainerDied","Data":"f18b15dc1d53b01e0eb276ed6e4c96330facd969155bf1a9010e39d94beab8aa"}
Sep 11 18:43:44.114816 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:44.114726    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f18b15dc1d53b01e0eb276ed6e4c96330facd969155bf1a9010e39d94beab8aa"
Sep 11 18:43:52.054889 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:52.054846    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:43:55.549107 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:55.549071    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"]
Sep 11 18:43:55.549541 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:55.549311    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4" podUID="06fa8c78-90b6-4dac-88e0-7dbc2a389d2f" containerName="payload-processing" containerID="cri-o://b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398" gracePeriod=30
Sep 11 18:43:55.714641 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:55.714605    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"]
Sep 11 18:43:55.719194 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:55.718590    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-999aa4-mfzzt"]
Sep 11 18:43:55.806645 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:55.806590    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:43:55.861881 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:55.861809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-plugins-config-volume\") pod \"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f\" (UID: \"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f\") "
Sep 11 18:43:55.862033 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:55.861936    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxmzm\" (UniqueName: \"kubernetes.io/projected/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-kube-api-access-fxmzm\") pod \"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f\" (UID: \"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f\") "
Sep 11 18:43:55.862228 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:55.862200    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06fa8c78-90b6-4dac-88e0-7dbc2a389d2f" (UID: "06fa8c78-90b6-4dac-88e0-7dbc2a389d2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:55.864105 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:55.864078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-kube-api-access-fxmzm" (OuterVolumeSpecName: "kube-api-access-fxmzm") pod "06fa8c78-90b6-4dac-88e0-7dbc2a389d2f" (UID: "06fa8c78-90b6-4dac-88e0-7dbc2a389d2f"). InnerVolumeSpecName "kube-api-access-fxmzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:55.962575 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:55.962551    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:55.962575 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:55.962574    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxmzm\" (UniqueName: \"kubernetes.io/projected/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f-kube-api-access-fxmzm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:43:56.170789 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:56.170758    2563 generic.go:356] "Generic (PLEG): container finished" podID="06fa8c78-90b6-4dac-88e0-7dbc2a389d2f" containerID="b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398" exitCode=0
Sep 11 18:43:56.170936 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:56.170796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4" event={"ID":"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f","Type":"ContainerDied","Data":"b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398"}
Sep 11 18:43:56.170936 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:56.170814    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"
Sep 11 18:43:56.170936 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:56.170858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4" event={"ID":"06fa8c78-90b6-4dac-88e0-7dbc2a389d2f","Type":"ContainerDied","Data":"f0ef1f5aea09f211baaff3473b8bf130583c12811613e2d59e3689a3c4313271"}
Sep 11 18:43:56.170936 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:56.170893    2563 scope.go:117] "RemoveContainer" containerID="b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398"
Sep 11 18:43:56.182568 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:56.182547    2563 scope.go:117] "RemoveContainer" containerID="b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398"
Sep 11 18:43:56.183090 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:43:56.183072    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398\": container with ID starting with b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398 not found: ID does not exist" containerID="b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398"
Sep 11 18:43:56.183153 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:56.183097    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398"} err="failed to get container status \"b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398\": rpc error: code = NotFound desc = could not find container \"b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398\": container with ID starting with b8c3301a37967246d8ead867e72c4144bcb50536033740a8962bd2a57d7b1398 not found: ID does not exist"
Sep 11 18:43:56.196785 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:56.196764    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"]
Sep 11 18:43:56.201624 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:56.201603    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-999aa4-8c5f8866c-nkzk4"]
Sep 11 18:43:57.113501 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:57.113473    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06fa8c78-90b6-4dac-88e0-7dbc2a389d2f" path="/var/lib/kubelet/pods/06fa8c78-90b6-4dac-88e0-7dbc2a389d2f/volumes"
Sep 11 18:43:57.113840 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:43:57.113788    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23e061db-483e-49a7-b0fe-fb735107d59b" path="/var/lib/kubelet/pods/23e061db-483e-49a7-b0fe-fb735107d59b/volumes"
Sep 11 18:44:02.485791 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.485757    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"]
Sep 11 18:44:02.486171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.486158    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23e061db-483e-49a7-b0fe-fb735107d59b" containerName="revoke-keys"
Sep 11 18:44:02.486223 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.486173    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23e061db-483e-49a7-b0fe-fb735107d59b" containerName="revoke-keys"
Sep 11 18:44:02.486223 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.486183    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06fa8c78-90b6-4dac-88e0-7dbc2a389d2f" containerName="payload-processing"
Sep 11 18:44:02.486223 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.486188    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="06fa8c78-90b6-4dac-88e0-7dbc2a389d2f" containerName="payload-processing"
Sep 11 18:44:02.486330 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.486244    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="06fa8c78-90b6-4dac-88e0-7dbc2a389d2f" containerName="payload-processing"
Sep 11 18:44:02.486330 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.486252    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="23e061db-483e-49a7-b0fe-fb735107d59b" containerName="revoke-keys"
Sep 11 18:44:02.490683 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.490666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"
Sep 11 18:44:02.493333 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.493313    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:44:02.497280 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.497258    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"]
Sep 11 18:44:02.615548 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.615523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/550ad7ab-964e-4aa6-82e4-8f40ed37c116-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k\" (UID: \"550ad7ab-964e-4aa6-82e4-8f40ed37c116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"
Sep 11 18:44:02.716436 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.716408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/550ad7ab-964e-4aa6-82e4-8f40ed37c116-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k\" (UID: \"550ad7ab-964e-4aa6-82e4-8f40ed37c116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"
Sep 11 18:44:02.716969 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.716947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/550ad7ab-964e-4aa6-82e4-8f40ed37c116-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k\" (UID: \"550ad7ab-964e-4aa6-82e4-8f40ed37c116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"
Sep 11 18:44:02.802119 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:02.802064    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"
Sep 11 18:44:03.096029 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:03.095992    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"]
Sep 11 18:44:03.097699 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:44:03.097668    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550ad7ab_964e_4aa6_82e4_8f40ed37c116.slice/crio-8d963b30b5f4e6ed380edc6393cdba5a2a6e6a27d1e261046680492768e0644d WatchSource:0}: Error finding container 8d963b30b5f4e6ed380edc6393cdba5a2a6e6a27d1e261046680492768e0644d: Status 404 returned error can't find the container with id 8d963b30b5f4e6ed380edc6393cdba5a2a6e6a27d1e261046680492768e0644d
Sep 11 18:44:03.214568 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:03.214540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k" event={"ID":"550ad7ab-964e-4aa6-82e4-8f40ed37c116","Type":"ContainerStarted","Data":"8d963b30b5f4e6ed380edc6393cdba5a2a6e6a27d1e261046680492768e0644d"}
Sep 11 18:44:04.220858 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:04.220826    2563 generic.go:356] "Generic (PLEG): container finished" podID="550ad7ab-964e-4aa6-82e4-8f40ed37c116" containerID="5bf34a74dac012ded014c4db741b08d42fa91ae67614d33b754d73cb09b2e264" exitCode=0
Sep 11 18:44:04.221225 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:04.220900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k" event={"ID":"550ad7ab-964e-4aa6-82e4-8f40ed37c116","Type":"ContainerDied","Data":"5bf34a74dac012ded014c4db741b08d42fa91ae67614d33b754d73cb09b2e264"}
Sep 11 18:44:05.358952 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:05.358927    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"
Sep 11 18:44:05.442841 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:05.442814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/550ad7ab-964e-4aa6-82e4-8f40ed37c116-maas-api-service-ca\") pod \"550ad7ab-964e-4aa6-82e4-8f40ed37c116\" (UID: \"550ad7ab-964e-4aa6-82e4-8f40ed37c116\") "
Sep 11 18:44:05.443152 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:05.443129    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550ad7ab-964e-4aa6-82e4-8f40ed37c116-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "550ad7ab-964e-4aa6-82e4-8f40ed37c116" (UID: "550ad7ab-964e-4aa6-82e4-8f40ed37c116"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:05.544002 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:05.543935    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/550ad7ab-964e-4aa6-82e4-8f40ed37c116-maas-api-service-ca\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:06.238174 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:06.238143    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"
Sep 11 18:44:06.238342 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:06.238139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k" event={"ID":"550ad7ab-964e-4aa6-82e4-8f40ed37c116","Type":"ContainerDied","Data":"8d963b30b5f4e6ed380edc6393cdba5a2a6e6a27d1e261046680492768e0644d"}
Sep 11 18:44:06.238342 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:06.238296    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d963b30b5f4e6ed380edc6393cdba5a2a6e6a27d1e261046680492768e0644d"
Sep 11 18:44:14.139989 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:14.139913    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"]
Sep 11 18:44:14.145668 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:14.145642    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"]
Sep 11 18:44:14.157477 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:14.157456    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"]
Sep 11 18:44:14.166740 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:14.166709    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"]
Sep 11 18:44:14.846579 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:14.846542    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"]
Sep 11 18:44:14.850065 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:14.850039    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-93d67f-jp29k"]
Sep 11 18:44:15.113541 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:15.113475    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="550ad7ab-964e-4aa6-82e4-8f40ed37c116" path="/var/lib/kubelet/pods/550ad7ab-964e-4aa6-82e4-8f40ed37c116/volumes"
Sep 11 18:44:18.467120 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:18.467088    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:44:29.159164 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.159103    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx" podUID="243ae99a-e8e9-4017-91e3-ff3edc7439a4" containerName="main" containerID="cri-o://eaeb43af684e83977014c22729574b88447e62c889d7eb15ea173262f039082a" gracePeriod=45
Sep 11 18:44:29.164182 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.164154    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g" podUID="ec596a1f-0f31-4f47-b039-fb46fbed08c5" containerName="main" containerID="cri-o://8ea79d0dad99522b3e48a35b33542bc760c15242208328030b09c1ffc6322cac" gracePeriod=45
Sep 11 18:44:29.177052 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.177010    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss" podUID="89933326-0e42-4955-b4fc-10fa4de028b0" containerName="main" containerID="cri-o://6bad9b7d0cd265cfc12a689ca1c3a3dfba0decc286feab6727ac1f6c29dfc680" gracePeriod=45
Sep 11 18:44:29.188728 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.188687    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm" podUID="8ae996dc-66d2-44b6-99c4-d66c080476f3" containerName="main" containerID="cri-o://faf8e4012737f51b5eefb620b0f90d99128027bffa6a5b922c60649a445026e0" gracePeriod=45
Sep 11 18:44:29.353535 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.353429    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ae996dc-66d2-44b6-99c4-d66c080476f3" containerID="faf8e4012737f51b5eefb620b0f90d99128027bffa6a5b922c60649a445026e0" exitCode=0
Sep 11 18:44:29.353535 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.353507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm" event={"ID":"8ae996dc-66d2-44b6-99c4-d66c080476f3","Type":"ContainerDied","Data":"faf8e4012737f51b5eefb620b0f90d99128027bffa6a5b922c60649a445026e0"}
Sep 11 18:44:29.355855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.355156    2563 generic.go:356] "Generic (PLEG): container finished" podID="89933326-0e42-4955-b4fc-10fa4de028b0" containerID="6bad9b7d0cd265cfc12a689ca1c3a3dfba0decc286feab6727ac1f6c29dfc680" exitCode=0
Sep 11 18:44:29.355855 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.355261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss" event={"ID":"89933326-0e42-4955-b4fc-10fa4de028b0","Type":"ContainerDied","Data":"6bad9b7d0cd265cfc12a689ca1c3a3dfba0decc286feab6727ac1f6c29dfc680"}
Sep 11 18:44:29.358143 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.357573    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec596a1f-0f31-4f47-b039-fb46fbed08c5" containerID="8ea79d0dad99522b3e48a35b33542bc760c15242208328030b09c1ffc6322cac" exitCode=0
Sep 11 18:44:29.358143 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.357754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g" event={"ID":"ec596a1f-0f31-4f47-b039-fb46fbed08c5","Type":"ContainerDied","Data":"8ea79d0dad99522b3e48a35b33542bc760c15242208328030b09c1ffc6322cac"}
Sep 11 18:44:29.361838 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.360599    2563 generic.go:356] "Generic (PLEG): container finished" podID="243ae99a-e8e9-4017-91e3-ff3edc7439a4" containerID="eaeb43af684e83977014c22729574b88447e62c889d7eb15ea173262f039082a" exitCode=0
Sep 11 18:44:29.361838 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.360646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx" event={"ID":"243ae99a-e8e9-4017-91e3-ff3edc7439a4","Type":"ContainerDied","Data":"eaeb43af684e83977014c22729574b88447e62c889d7eb15ea173262f039082a"}
Sep 11 18:44:29.475367 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.475342    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:44:29.508099 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.508071    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:44:29.512564 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.512544    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:44:29.519644 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.519626    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:44:29.638149 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tls-certs\") pod \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") "
Sep 11 18:44:29.638340 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638160    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-home\") pod \"8ae996dc-66d2-44b6-99c4-d66c080476f3\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") "
Sep 11 18:44:29.638340 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-tmp-dir\") pod \"89933326-0e42-4955-b4fc-10fa4de028b0\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") "
Sep 11 18:44:29.638340 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-tmp-dir\") pod \"8ae996dc-66d2-44b6-99c4-d66c080476f3\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") "
Sep 11 18:44:29.638340 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638241    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-model-cache\") pod \"89933326-0e42-4955-b4fc-10fa4de028b0\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") "
Sep 11 18:44:29.638340 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tls-certs\") pod \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") "
Sep 11 18:44:29.638340 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638306    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-dshm\") pod \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") "
Sep 11 18:44:29.638340 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-model-cache\") pod \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638363    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn95k\" (UniqueName: \"kubernetes.io/projected/243ae99a-e8e9-4017-91e3-ff3edc7439a4-kube-api-access-wn95k\") pod \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638420    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89933326-0e42-4955-b4fc-10fa4de028b0-tls-certs\") pod \"89933326-0e42-4955-b4fc-10fa4de028b0\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638449    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-home\") pod \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ae996dc-66d2-44b6-99c4-d66c080476f3" (UID: "8ae996dc-66d2-44b6-99c4-d66c080476f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638486    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-dshm\") pod \"8ae996dc-66d2-44b6-99c4-d66c080476f3\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638552    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae996dc-66d2-44b6-99c4-d66c080476f3-tls-certs\") pod \"8ae996dc-66d2-44b6-99c4-d66c080476f3\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638577    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-dshm\") pod \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tmp-dir\") pod \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-home\") pod \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\" (UID: \"243ae99a-e8e9-4017-91e3-ff3edc7439a4\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638673    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxkpk\" (UniqueName: \"kubernetes.io/projected/ec596a1f-0f31-4f47-b039-fb46fbed08c5-kube-api-access-sxkpk\") pod \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-model-cache\") pod \"8ae996dc-66d2-44b6-99c4-d66c080476f3\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638735    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tmp-dir\") pod \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") "
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638462    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-home" (OuterVolumeSpecName: "home") pod "8ae996dc-66d2-44b6-99c4-d66c080476f3" (UID: "8ae996dc-66d2-44b6-99c4-d66c080476f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638507    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89933326-0e42-4955-b4fc-10fa4de028b0" (UID: "89933326-0e42-4955-b4fc-10fa4de028b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "89933326-0e42-4955-b4fc-10fa4de028b0" (UID: "89933326-0e42-4955-b4fc-10fa4de028b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.638770 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638721    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "243ae99a-e8e9-4017-91e3-ff3edc7439a4" (UID: "243ae99a-e8e9-4017-91e3-ff3edc7439a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638730    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-home" (OuterVolumeSpecName: "home") pod "ec596a1f-0f31-4f47-b039-fb46fbed08c5" (UID: "ec596a1f-0f31-4f47-b039-fb46fbed08c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638792    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92xmw\" (UniqueName: \"kubernetes.io/projected/89933326-0e42-4955-b4fc-10fa4de028b0-kube-api-access-92xmw\") pod \"89933326-0e42-4955-b4fc-10fa4de028b0\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") "
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-model-cache\") pod \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\" (UID: \"ec596a1f-0f31-4f47-b039-fb46fbed08c5\") "
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-dshm\") pod \"89933326-0e42-4955-b4fc-10fa4de028b0\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") "
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638888    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-home\") pod \"89933326-0e42-4955-b4fc-10fa4de028b0\" (UID: \"89933326-0e42-4955-b4fc-10fa4de028b0\") "
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.638916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d64j7\" (UniqueName: \"kubernetes.io/projected/8ae996dc-66d2-44b6-99c4-d66c080476f3-kube-api-access-d64j7\") pod \"8ae996dc-66d2-44b6-99c4-d66c080476f3\" (UID: \"8ae996dc-66d2-44b6-99c4-d66c080476f3\") "
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.639276    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.639294    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.639306    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.639319    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.639331    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.639343    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.639482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec596a1f-0f31-4f47-b039-fb46fbed08c5" (UID: "ec596a1f-0f31-4f47-b039-fb46fbed08c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.639635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.639485    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-home" (OuterVolumeSpecName: "home") pod "243ae99a-e8e9-4017-91e3-ff3edc7439a4" (UID: "243ae99a-e8e9-4017-91e3-ff3edc7439a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.640714 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.639692    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "243ae99a-e8e9-4017-91e3-ff3edc7439a4" (UID: "243ae99a-e8e9-4017-91e3-ff3edc7439a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.640714 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.640111    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ae996dc-66d2-44b6-99c4-d66c080476f3" (UID: "8ae996dc-66d2-44b6-99c4-d66c080476f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.640714 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.640307    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec596a1f-0f31-4f47-b039-fb46fbed08c5" (UID: "ec596a1f-0f31-4f47-b039-fb46fbed08c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.641248 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.641217    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec596a1f-0f31-4f47-b039-fb46fbed08c5" (UID: "ec596a1f-0f31-4f47-b039-fb46fbed08c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:29.641354 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.641313    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-home" (OuterVolumeSpecName: "home") pod "89933326-0e42-4955-b4fc-10fa4de028b0" (UID: "89933326-0e42-4955-b4fc-10fa4de028b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.641630 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.641605    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89933326-0e42-4955-b4fc-10fa4de028b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89933326-0e42-4955-b4fc-10fa4de028b0" (UID: "89933326-0e42-4955-b4fc-10fa4de028b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:29.641722 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.641680    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-dshm" (OuterVolumeSpecName: "dshm") pod "243ae99a-e8e9-4017-91e3-ff3edc7439a4" (UID: "243ae99a-e8e9-4017-91e3-ff3edc7439a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.641789 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.641775    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-dshm" (OuterVolumeSpecName: "dshm") pod "ec596a1f-0f31-4f47-b039-fb46fbed08c5" (UID: "ec596a1f-0f31-4f47-b039-fb46fbed08c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.642143 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.642104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "243ae99a-e8e9-4017-91e3-ff3edc7439a4" (UID: "243ae99a-e8e9-4017-91e3-ff3edc7439a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:29.642243 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.642146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae996dc-66d2-44b6-99c4-d66c080476f3-kube-api-access-d64j7" (OuterVolumeSpecName: "kube-api-access-d64j7") pod "8ae996dc-66d2-44b6-99c4-d66c080476f3" (UID: "8ae996dc-66d2-44b6-99c4-d66c080476f3"). InnerVolumeSpecName "kube-api-access-d64j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:29.642574 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.642552    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243ae99a-e8e9-4017-91e3-ff3edc7439a4-kube-api-access-wn95k" (OuterVolumeSpecName: "kube-api-access-wn95k") pod "243ae99a-e8e9-4017-91e3-ff3edc7439a4" (UID: "243ae99a-e8e9-4017-91e3-ff3edc7439a4"). InnerVolumeSpecName "kube-api-access-wn95k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:29.642765 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.642750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-dshm" (OuterVolumeSpecName: "dshm") pod "8ae996dc-66d2-44b6-99c4-d66c080476f3" (UID: "8ae996dc-66d2-44b6-99c4-d66c080476f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.642956 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.642940    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-dshm" (OuterVolumeSpecName: "dshm") pod "89933326-0e42-4955-b4fc-10fa4de028b0" (UID: "89933326-0e42-4955-b4fc-10fa4de028b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:29.643271 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.643252    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ae996dc-66d2-44b6-99c4-d66c080476f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ae996dc-66d2-44b6-99c4-d66c080476f3" (UID: "8ae996dc-66d2-44b6-99c4-d66c080476f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:29.643700 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.643675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec596a1f-0f31-4f47-b039-fb46fbed08c5-kube-api-access-sxkpk" (OuterVolumeSpecName: "kube-api-access-sxkpk") pod "ec596a1f-0f31-4f47-b039-fb46fbed08c5" (UID: "ec596a1f-0f31-4f47-b039-fb46fbed08c5"). InnerVolumeSpecName "kube-api-access-sxkpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:29.643804 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.643764    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89933326-0e42-4955-b4fc-10fa4de028b0-kube-api-access-92xmw" (OuterVolumeSpecName: "kube-api-access-92xmw") pod "89933326-0e42-4955-b4fc-10fa4de028b0" (UID: "89933326-0e42-4955-b4fc-10fa4de028b0"). InnerVolumeSpecName "kube-api-access-92xmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:29.739856 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739798    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxkpk\" (UniqueName: \"kubernetes.io/projected/ec596a1f-0f31-4f47-b039-fb46fbed08c5-kube-api-access-sxkpk\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.739856 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739821    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.739856 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739831    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.739856 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739839    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92xmw\" (UniqueName: \"kubernetes.io/projected/89933326-0e42-4955-b4fc-10fa4de028b0-kube-api-access-92xmw\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.739856 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739848    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.739856 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739855    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739862    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89933326-0e42-4955-b4fc-10fa4de028b0-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739869    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d64j7\" (UniqueName: \"kubernetes.io/projected/8ae996dc-66d2-44b6-99c4-d66c080476f3-kube-api-access-d64j7\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739876    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec596a1f-0f31-4f47-b039-fb46fbed08c5-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739885    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739892    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739899    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn95k\" (UniqueName: \"kubernetes.io/projected/243ae99a-e8e9-4017-91e3-ff3edc7439a4-kube-api-access-wn95k\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739905    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89933326-0e42-4955-b4fc-10fa4de028b0-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739912    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae996dc-66d2-44b6-99c4-d66c080476f3-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739920    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae996dc-66d2-44b6-99c4-d66c080476f3-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739926    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec596a1f-0f31-4f47-b039-fb46fbed08c5-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739933    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:29.740108 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:29.739940    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243ae99a-e8e9-4017-91e3-ff3edc7439a4-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:30.368297 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.368266    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"
Sep 11 18:44:30.368297 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.368274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss" event={"ID":"89933326-0e42-4955-b4fc-10fa4de028b0","Type":"ContainerDied","Data":"6a209690b1053f11e7b38b8b1fe3ce209350258114724a2c92e40f16755e4bd8"}
Sep 11 18:44:30.368815 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.368330    2563 scope.go:117] "RemoveContainer" containerID="6bad9b7d0cd265cfc12a689ca1c3a3dfba0decc286feab6727ac1f6c29dfc680"
Sep 11 18:44:30.370211 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.369985    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"
Sep 11 18:44:30.370381 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.370017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g" event={"ID":"ec596a1f-0f31-4f47-b039-fb46fbed08c5","Type":"ContainerDied","Data":"4b0b8e1e13960c15057716fa6ca19a5dd5eff5626cd5107c581edee66af74790"}
Sep 11 18:44:30.372036 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.371995    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"
Sep 11 18:44:30.372147 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.372010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx" event={"ID":"243ae99a-e8e9-4017-91e3-ff3edc7439a4","Type":"ContainerDied","Data":"b23c8025c6eb9163de885d4a9338cff15eb92094a0a4949dd794c2d10af5dd29"}
Sep 11 18:44:30.373669 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.373628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm" event={"ID":"8ae996dc-66d2-44b6-99c4-d66c080476f3","Type":"ContainerDied","Data":"c77e652ad8f46b1b03cad84c0311ef162c6885e7284e9b79a2bdccc5ab19daa2"}
Sep 11 18:44:30.373669 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.373664    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"
Sep 11 18:44:30.379742 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.379724    2563 scope.go:117] "RemoveContainer" containerID="8ea79d0dad99522b3e48a35b33542bc760c15242208328030b09c1ffc6322cac"
Sep 11 18:44:30.388703 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.388685    2563 scope.go:117] "RemoveContainer" containerID="eaeb43af684e83977014c22729574b88447e62c889d7eb15ea173262f039082a"
Sep 11 18:44:30.396260 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.396246    2563 scope.go:117] "RemoveContainer" containerID="faf8e4012737f51b5eefb620b0f90d99128027bffa6a5b922c60649a445026e0"
Sep 11 18:44:30.396482 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.396465    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"]
Sep 11 18:44:30.400983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.400962    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/premium-w3-c8b274-kserve-9bcfbb789-kl5ss"]
Sep 11 18:44:30.410953 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.410930    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"]
Sep 11 18:44:30.414712 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.414692    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/simulator-w3-c8b274-kserve-59dc6c4d56-xx9vm"]
Sep 11 18:44:30.425226 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.425166    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"]
Sep 11 18:44:30.431202 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.431180    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-w3-c8b274-kserve-58dbc55f48-tzx6g"]
Sep 11 18:44:30.441526 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.441506    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"]
Sep 11 18:44:30.447168 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:30.447140    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c8b274/distinct-2-w3-c8b274-kserve-6b97d7f4c-6mcxx"]
Sep 11 18:44:31.113438 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:31.113401    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="243ae99a-e8e9-4017-91e3-ff3edc7439a4" path="/var/lib/kubelet/pods/243ae99a-e8e9-4017-91e3-ff3edc7439a4/volumes"
Sep 11 18:44:31.113818 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:31.113804    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89933326-0e42-4955-b4fc-10fa4de028b0" path="/var/lib/kubelet/pods/89933326-0e42-4955-b4fc-10fa4de028b0/volumes"
Sep 11 18:44:31.114149 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:31.114137    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ae996dc-66d2-44b6-99c4-d66c080476f3" path="/var/lib/kubelet/pods/8ae996dc-66d2-44b6-99c4-d66c080476f3/volumes"
Sep 11 18:44:31.114805 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:31.114789    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec596a1f-0f31-4f47-b039-fb46fbed08c5" path="/var/lib/kubelet/pods/ec596a1f-0f31-4f47-b039-fb46fbed08c5/volumes"
Sep 11 18:44:33.200009 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.199978    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"]
Sep 11 18:44:33.200428 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.200176    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s" podUID="52a16ae8-ad4a-4222-b766-f964212f32b8" containerName="maas-api" containerID="cri-o://8a5db80c0ccb7eaf8cac54154806dfcecb8f7705029e8a3ba4f170b70a99670b" gracePeriod=30
Sep 11 18:44:33.405068 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.405039    2563 generic.go:356] "Generic (PLEG): container finished" podID="52a16ae8-ad4a-4222-b766-f964212f32b8" containerID="8a5db80c0ccb7eaf8cac54154806dfcecb8f7705029e8a3ba4f170b70a99670b" exitCode=0
Sep 11 18:44:33.405232 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.405105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s" event={"ID":"52a16ae8-ad4a-4222-b766-f964212f32b8","Type":"ContainerDied","Data":"8a5db80c0ccb7eaf8cac54154806dfcecb8f7705029e8a3ba4f170b70a99670b"}
Sep 11 18:44:33.449797 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.449772    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:44:33.469464 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.469409    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls\") pod \"52a16ae8-ad4a-4222-b766-f964212f32b8\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") "
Sep 11 18:44:33.469680 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.469557    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk8gk\" (UniqueName: \"kubernetes.io/projected/52a16ae8-ad4a-4222-b766-f964212f32b8-kube-api-access-vk8gk\") pod \"52a16ae8-ad4a-4222-b766-f964212f32b8\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") "
Sep 11 18:44:33.469792 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.469588    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a16ae8-ad4a-4222-b766-f964212f32b8-openshift-service-ca\") pod \"52a16ae8-ad4a-4222-b766-f964212f32b8\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") "
Sep 11 18:44:33.469992 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.469974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-metrics-certs\") pod \"52a16ae8-ad4a-4222-b766-f964212f32b8\" (UID: \"52a16ae8-ad4a-4222-b766-f964212f32b8\") "
Sep 11 18:44:33.470234 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.469986    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a16ae8-ad4a-4222-b766-f964212f32b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52a16ae8-ad4a-4222-b766-f964212f32b8" (UID: "52a16ae8-ad4a-4222-b766-f964212f32b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:33.470572 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.470552    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a16ae8-ad4a-4222-b766-f964212f32b8-openshift-service-ca\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:33.472442 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.472409    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52a16ae8-ad4a-4222-b766-f964212f32b8" (UID: "52a16ae8-ad4a-4222-b766-f964212f32b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:33.472776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.472743    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a16ae8-ad4a-4222-b766-f964212f32b8-kube-api-access-vk8gk" (OuterVolumeSpecName: "kube-api-access-vk8gk") pod "52a16ae8-ad4a-4222-b766-f964212f32b8" (UID: "52a16ae8-ad4a-4222-b766-f964212f32b8"). InnerVolumeSpecName "kube-api-access-vk8gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:33.472939 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.472911    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52a16ae8-ad4a-4222-b766-f964212f32b8" (UID: "52a16ae8-ad4a-4222-b766-f964212f32b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:33.571772 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.571737    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk8gk\" (UniqueName: \"kubernetes.io/projected/52a16ae8-ad4a-4222-b766-f964212f32b8-kube-api-access-vk8gk\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:33.571772 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.571772    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-metrics-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:33.571772 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:33.571782    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a16ae8-ad4a-4222-b766-f964212f32b8-maas-api-tls\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:44:34.411301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:34.411273    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"
Sep 11 18:44:34.411301 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:34.411283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s" event={"ID":"52a16ae8-ad4a-4222-b766-f964212f32b8","Type":"ContainerDied","Data":"c984749ed977a2fe550b5189aa9aaf01f89b222616c1b260c3e6cbad24142263"}
Sep 11 18:44:34.411793 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:34.411334    2563 scope.go:117] "RemoveContainer" containerID="8a5db80c0ccb7eaf8cac54154806dfcecb8f7705029e8a3ba4f170b70a99670b"
Sep 11 18:44:34.432292 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:34.432271    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"]
Sep 11 18:44:34.437486 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:34.437467    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e6454f-75458c867d-jf77s"]
Sep 11 18:44:35.113454 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:35.113421    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52a16ae8-ad4a-4222-b766-f964212f32b8" path="/var/lib/kubelet/pods/52a16ae8-ad4a-4222-b766-f964212f32b8/volumes"
Sep 11 18:44:38.143477 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:38.143430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:44:51.665050 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:51.665014    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:44:58.554034 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.553997    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"]
Sep 11 18:44:58.554571 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554552    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89933326-0e42-4955-b4fc-10fa4de028b0" containerName="main"
Sep 11 18:44:58.554651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554574    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="89933326-0e42-4955-b4fc-10fa4de028b0" containerName="main"
Sep 11 18:44:58.554651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554593    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="243ae99a-e8e9-4017-91e3-ff3edc7439a4" containerName="main"
Sep 11 18:44:58.554651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554602    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="243ae99a-e8e9-4017-91e3-ff3edc7439a4" containerName="main"
Sep 11 18:44:58.554651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554617    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec596a1f-0f31-4f47-b039-fb46fbed08c5" containerName="main"
Sep 11 18:44:58.554651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554625    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec596a1f-0f31-4f47-b039-fb46fbed08c5" containerName="main"
Sep 11 18:44:58.554651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554639    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a16ae8-ad4a-4222-b766-f964212f32b8" containerName="maas-api"
Sep 11 18:44:58.554651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554647    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a16ae8-ad4a-4222-b766-f964212f32b8" containerName="maas-api"
Sep 11 18:44:58.554987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554663    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550ad7ab-964e-4aa6-82e4-8f40ed37c116" containerName="revoke-keys"
Sep 11 18:44:58.554987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554671    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="550ad7ab-964e-4aa6-82e4-8f40ed37c116" containerName="revoke-keys"
Sep 11 18:44:58.554987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554689    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae996dc-66d2-44b6-99c4-d66c080476f3" containerName="main"
Sep 11 18:44:58.554987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554696    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae996dc-66d2-44b6-99c4-d66c080476f3" containerName="main"
Sep 11 18:44:58.554987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554811    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="89933326-0e42-4955-b4fc-10fa4de028b0" containerName="main"
Sep 11 18:44:58.554987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554824    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a16ae8-ad4a-4222-b766-f964212f32b8" containerName="maas-api"
Sep 11 18:44:58.554987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554833    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae996dc-66d2-44b6-99c4-d66c080476f3" containerName="main"
Sep 11 18:44:58.554987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554843    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec596a1f-0f31-4f47-b039-fb46fbed08c5" containerName="main"
Sep 11 18:44:58.554987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554854    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="243ae99a-e8e9-4017-91e3-ff3edc7439a4" containerName="main"
Sep 11 18:44:58.554987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.554862    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="550ad7ab-964e-4aa6-82e4-8f40ed37c116" containerName="revoke-keys"
Sep 11 18:44:58.558053 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.558033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.561576 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.561546    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-ec9107\"/\"openshift-service-ca.crt\""
Sep 11 18:44:58.561681 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.561547    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-ec9107\"/\"kube-root-ca.crt\""
Sep 11 18:44:58.561681 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.561547    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-ec9107\"/\"default-dockercfg-59fq9\""
Sep 11 18:44:58.561681 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.561627    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-ec9107\"/\"sub-test-model-ec9107-kserve-self-signed-certs\""
Sep 11 18:44:58.569331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.569309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"]
Sep 11 18:44:58.671110 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.671081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-home\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.671228 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.671151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-tmp-dir\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.671228 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.671180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bml8l\" (UniqueName: \"kubernetes.io/projected/c32de7bf-cd55-47fb-9938-6471c3958cb5-kube-api-access-bml8l\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.671336 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.671227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-dshm\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.671336 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.671262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32de7bf-cd55-47fb-9938-6471c3958cb5-tls-certs\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.671442 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.671348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-model-cache\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.772543 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.772519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-model-cache\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.772635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.772549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-home\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.772635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.772589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-tmp-dir\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.772635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.772604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bml8l\" (UniqueName: \"kubernetes.io/projected/c32de7bf-cd55-47fb-9938-6471c3958cb5-kube-api-access-bml8l\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.772635 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.772631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-dshm\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.772823 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.772664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32de7bf-cd55-47fb-9938-6471c3958cb5-tls-certs\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.773013 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.772988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-model-cache\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.773085 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.773012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-tmp-dir\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.773085 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.773037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-home\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.774989 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.774959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-dshm\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.775193 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.775172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32de7bf-cd55-47fb-9938-6471c3958cb5-tls-certs\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.781023 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.781005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bml8l\" (UniqueName: \"kubernetes.io/projected/c32de7bf-cd55-47fb-9938-6471c3958cb5-kube-api-access-bml8l\") pod \"sub-test-model-ec9107-kserve-787dd8ffbf-cjp26\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:58.866619 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:58.866563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:44:59.142324 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:59.142302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"]
Sep 11 18:44:59.145059 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:44:59.145029    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc32de7bf_cd55_47fb_9938_6471c3958cb5.slice/crio-f420e8b3f66d0e60128279b729912432f182b598fe14f066bb838b4223c6de14 WatchSource:0}: Error finding container f420e8b3f66d0e60128279b729912432f182b598fe14f066bb838b4223c6de14: Status 404 returned error can't find the container with id f420e8b3f66d0e60128279b729912432f182b598fe14f066bb838b4223c6de14
Sep 11 18:44:59.530074 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:59.530037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26" event={"ID":"c32de7bf-cd55-47fb-9938-6471c3958cb5","Type":"ContainerStarted","Data":"8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252"}
Sep 11 18:44:59.530231 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:59.530092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26" event={"ID":"c32de7bf-cd55-47fb-9938-6471c3958cb5","Type":"ContainerStarted","Data":"f420e8b3f66d0e60128279b729912432f182b598fe14f066bb838b4223c6de14"}
Sep 11 18:44:59.549424 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:44:59.549367    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26" podStartSLOduration=1.549355069 podStartE2EDuration="1.549355069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:44:59.547960607 +0000 UTC m=+1754.942475657" watchObservedRunningTime="2026-09-11 18:44:59.549355069 +0000 UTC m=+1754.943870092"
Sep 11 18:45:00.136561 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:00.136527    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz"]
Sep 11 18:45:00.139628 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:00.139609    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz"
Sep 11 18:45:00.143147 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:00.143130    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:45:00.149262 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:00.149242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz"
Sep 11 18:45:00.150422 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:00.150399    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz"]
Sep 11 18:45:00.418718 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:00.418696    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz"]
Sep 11 18:45:00.421127 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:45:00.421091    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1963cbca_2880_4a0c_926a_7f3da46e68b9.slice/crio-6c72d57b8d4446b57042b3d4f2f909c51e511eb57b762fb7757baf2346e86fb4 WatchSource:0}: Error finding container 6c72d57b8d4446b57042b3d4f2f909c51e511eb57b762fb7757baf2346e86fb4: Status 404 returned error can't find the container with id 6c72d57b8d4446b57042b3d4f2f909c51e511eb57b762fb7757baf2346e86fb4
Sep 11 18:45:00.536071 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:00.536045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz" event={"ID":"1963cbca-2880-4a0c-926a-7f3da46e68b9","Type":"ContainerStarted","Data":"6c72d57b8d4446b57042b3d4f2f909c51e511eb57b762fb7757baf2346e86fb4"}
Sep 11 18:45:01.542605 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:01.542540    2563 generic.go:356] "Generic (PLEG): container finished" podID="1963cbca-2880-4a0c-926a-7f3da46e68b9" containerID="84eded03f613aeffdaa0788a2af291ff79ea82ccaf3082fb1a8743b65254378f" exitCode=0
Sep 11 18:45:01.542884 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:01.542614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz" event={"ID":"1963cbca-2880-4a0c-926a-7f3da46e68b9","Type":"ContainerDied","Data":"84eded03f613aeffdaa0788a2af291ff79ea82ccaf3082fb1a8743b65254378f"}
Sep 11 18:45:02.679595 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:02.679574    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz"
Sep 11 18:45:02.761646 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:02.761609    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:45:03.553943 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:03.553914    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz"
Sep 11 18:45:03.553943 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:03.553934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz" event={"ID":"1963cbca-2880-4a0c-926a-7f3da46e68b9","Type":"ContainerDied","Data":"6c72d57b8d4446b57042b3d4f2f909c51e511eb57b762fb7757baf2346e86fb4"}
Sep 11 18:45:03.554183 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:03.553983    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c72d57b8d4446b57042b3d4f2f909c51e511eb57b762fb7757baf2346e86fb4"
Sep 11 18:45:08.867630 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:08.867595    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:45:08.867997 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:08.867642    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:45:08.879840 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:08.879813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:45:09.593578 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:09.593545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:45:42.172053 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:42.171979    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:45:45.186274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:45.186240    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:45:45.192656 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:45.192629    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:45:45.194260 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:45.194238    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:45:45.200089 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:45.200072    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:45:57.654224 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:45:57.654186    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:46:10.357118 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:10.357080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:46:20.433990 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:20.433951    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"]
Sep 11 18:46:24.449009 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:24.448973    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:46:28.252764 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.252724    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"]
Sep 11 18:46:28.253252 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.253232    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1963cbca-2880-4a0c-926a-7f3da46e68b9" containerName="cleanup"
Sep 11 18:46:28.253327 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.253254    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1963cbca-2880-4a0c-926a-7f3da46e68b9" containerName="cleanup"
Sep 11 18:46:28.253428 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.253405    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1963cbca-2880-4a0c-926a-7f3da46e68b9" containerName="cleanup"
Sep 11 18:46:28.256504 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.256481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.259090 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.259064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-ec9107\"/\"rate-test-model-ec9107-kserve-self-signed-certs\""
Sep 11 18:46:28.266560 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.266525    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"]
Sep 11 18:46:28.382612 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.382575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d26n\" (UniqueName: \"kubernetes.io/projected/8437182a-56de-4ab8-91de-3be01bf8fdc2-kube-api-access-6d26n\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.382768 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.382627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-dshm\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.382768 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.382672    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-model-cache\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.382768 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.382698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-home\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.382768 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.382714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8437182a-56de-4ab8-91de-3be01bf8fdc2-tls-certs\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.382910 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.382782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-tmp-dir\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.483703 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.483672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-model-cache\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.483861 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.483715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-home\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.483861 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.483732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8437182a-56de-4ab8-91de-3be01bf8fdc2-tls-certs\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.483861 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.483773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-tmp-dir\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.483861 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.483805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d26n\" (UniqueName: \"kubernetes.io/projected/8437182a-56de-4ab8-91de-3be01bf8fdc2-kube-api-access-6d26n\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.483861 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.483839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-dshm\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.484121 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.484102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-model-cache\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.484180 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.484132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-home\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.484219 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.484200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-tmp-dir\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.485919 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.485898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-dshm\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.486088 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.486071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8437182a-56de-4ab8-91de-3be01bf8fdc2-tls-certs\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.492742 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.492712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d26n\" (UniqueName: \"kubernetes.io/projected/8437182a-56de-4ab8-91de-3be01bf8fdc2-kube-api-access-6d26n\") pod \"rate-test-model-ec9107-kserve-87949794c-b57wh\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.567768 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.567715    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:28.836854 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.836786    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"]
Sep 11 18:46:28.840036 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:46:28.840001    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8437182a_56de_4ab8_91de_3be01bf8fdc2.slice/crio-29ae77923802cffda47738bdfbead0f09f4cebc127dc7a0567820d91cb8e88ce WatchSource:0}: Error finding container 29ae77923802cffda47738bdfbead0f09f4cebc127dc7a0567820d91cb8e88ce: Status 404 returned error can't find the container with id 29ae77923802cffda47738bdfbead0f09f4cebc127dc7a0567820d91cb8e88ce
Sep 11 18:46:28.960033 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.960002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh" event={"ID":"8437182a-56de-4ab8-91de-3be01bf8fdc2","Type":"ContainerStarted","Data":"bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4"}
Sep 11 18:46:28.960160 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.960046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh" event={"ID":"8437182a-56de-4ab8-91de-3be01bf8fdc2","Type":"ContainerStarted","Data":"29ae77923802cffda47738bdfbead0f09f4cebc127dc7a0567820d91cb8e88ce"}
Sep 11 18:46:28.977306 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:28.977264    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh" podStartSLOduration=0.977250157 podStartE2EDuration="977.250157ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:46:28.975665029 +0000 UTC m=+1844.370180074" watchObservedRunningTime="2026-09-11 18:46:28.977250157 +0000 UTC m=+1844.371765180"
Sep 11 18:46:35.453482 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.453424    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26" podUID="c32de7bf-cd55-47fb-9938-6471c3958cb5" containerName="main" containerID="cri-o://8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252" gracePeriod=45
Sep 11 18:46:35.708648 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.708598    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:46:35.733745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.733723    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-tmp-dir\") pod \"c32de7bf-cd55-47fb-9938-6471c3958cb5\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") "
Sep 11 18:46:35.733880 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.733797    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bml8l\" (UniqueName: \"kubernetes.io/projected/c32de7bf-cd55-47fb-9938-6471c3958cb5-kube-api-access-bml8l\") pod \"c32de7bf-cd55-47fb-9938-6471c3958cb5\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") "
Sep 11 18:46:35.733880 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.733827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-dshm\") pod \"c32de7bf-cd55-47fb-9938-6471c3958cb5\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") "
Sep 11 18:46:35.733880 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.733843    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32de7bf-cd55-47fb-9938-6471c3958cb5-tls-certs\") pod \"c32de7bf-cd55-47fb-9938-6471c3958cb5\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") "
Sep 11 18:46:35.733880 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.733866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-home\") pod \"c32de7bf-cd55-47fb-9938-6471c3958cb5\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") "
Sep 11 18:46:35.734090 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.733904    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-model-cache\") pod \"c32de7bf-cd55-47fb-9938-6471c3958cb5\" (UID: \"c32de7bf-cd55-47fb-9938-6471c3958cb5\") "
Sep 11 18:46:35.734090 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.733989    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c32de7bf-cd55-47fb-9938-6471c3958cb5" (UID: "c32de7bf-cd55-47fb-9938-6471c3958cb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:35.734202 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.734176    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:46:35.734259 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.734198    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-home" (OuterVolumeSpecName: "home") pod "c32de7bf-cd55-47fb-9938-6471c3958cb5" (UID: "c32de7bf-cd55-47fb-9938-6471c3958cb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:35.734259 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.734218    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c32de7bf-cd55-47fb-9938-6471c3958cb5" (UID: "c32de7bf-cd55-47fb-9938-6471c3958cb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:35.736054 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.736028    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32de7bf-cd55-47fb-9938-6471c3958cb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c32de7bf-cd55-47fb-9938-6471c3958cb5" (UID: "c32de7bf-cd55-47fb-9938-6471c3958cb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:46:35.736054 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.736042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-dshm" (OuterVolumeSpecName: "dshm") pod "c32de7bf-cd55-47fb-9938-6471c3958cb5" (UID: "c32de7bf-cd55-47fb-9938-6471c3958cb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:35.736181 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.736052    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32de7bf-cd55-47fb-9938-6471c3958cb5-kube-api-access-bml8l" (OuterVolumeSpecName: "kube-api-access-bml8l") pod "c32de7bf-cd55-47fb-9938-6471c3958cb5" (UID: "c32de7bf-cd55-47fb-9938-6471c3958cb5"). InnerVolumeSpecName "kube-api-access-bml8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:46:35.834941 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.834920    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bml8l\" (UniqueName: \"kubernetes.io/projected/c32de7bf-cd55-47fb-9938-6471c3958cb5-kube-api-access-bml8l\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:46:35.834941 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.834939    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:46:35.835070 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.834949    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32de7bf-cd55-47fb-9938-6471c3958cb5-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:46:35.835070 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.834958    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:46:35.835070 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.834966    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32de7bf-cd55-47fb-9938-6471c3958cb5-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:46:35.998024 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.997967    2563 generic.go:356] "Generic (PLEG): container finished" podID="c32de7bf-cd55-47fb-9938-6471c3958cb5" containerID="8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252" exitCode=0
Sep 11 18:46:35.998024 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.998008    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"
Sep 11 18:46:35.998146 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.998026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26" event={"ID":"c32de7bf-cd55-47fb-9938-6471c3958cb5","Type":"ContainerDied","Data":"8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252"}
Sep 11 18:46:35.998146 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.998066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26" event={"ID":"c32de7bf-cd55-47fb-9938-6471c3958cb5","Type":"ContainerDied","Data":"f420e8b3f66d0e60128279b729912432f182b598fe14f066bb838b4223c6de14"}
Sep 11 18:46:35.998146 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:35.998085    2563 scope.go:117] "RemoveContainer" containerID="8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252"
Sep 11 18:46:36.007055 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:36.007039    2563 scope.go:117] "RemoveContainer" containerID="8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252"
Sep 11 18:46:36.007331 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:46:36.007312    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252\": container with ID starting with 8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252 not found: ID does not exist" containerID="8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252"
Sep 11 18:46:36.007404 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:36.007339    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252"} err="failed to get container status \"8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252\": rpc error: code = NotFound desc = could not find container \"8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252\": container with ID starting with 8ed187769fc6336165dcac9935c7b05a0e036b0a72dc6f876b77507bec258252 not found: ID does not exist"
Sep 11 18:46:36.018943 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:36.018921    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"]
Sep 11 18:46:36.022527 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:36.022505    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/sub-test-model-ec9107-kserve-787dd8ffbf-cjp26"]
Sep 11 18:46:37.112775 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:37.112740    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c32de7bf-cd55-47fb-9938-6471c3958cb5" path="/var/lib/kubelet/pods/c32de7bf-cd55-47fb-9938-6471c3958cb5/volumes"
Sep 11 18:46:37.546924 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:37.546892    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:46:38.568341 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:38.568307    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:38.568721 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:38.568354    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:38.580767 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:38.580744    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:39.025860 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:39.025835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:46:56.772268 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:46:56.772232    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:47:18.551394 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:18.551294    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:47:27.844418 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:27.844362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:47:34.761294 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:34.761257    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"]
Sep 11 18:47:38.250768 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:38.250733    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:47:49.780066 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:49.780015    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh" podUID="8437182a-56de-4ab8-91de-3be01bf8fdc2" containerName="main" containerID="cri-o://bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4" gracePeriod=45
Sep 11 18:47:50.037361 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.037299    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:47:50.198643 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.198618    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d26n\" (UniqueName: \"kubernetes.io/projected/8437182a-56de-4ab8-91de-3be01bf8fdc2-kube-api-access-6d26n\") pod \"8437182a-56de-4ab8-91de-3be01bf8fdc2\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") "
Sep 11 18:47:50.198764 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.198677    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-dshm\") pod \"8437182a-56de-4ab8-91de-3be01bf8fdc2\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") "
Sep 11 18:47:50.198764 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.198699    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-tmp-dir\") pod \"8437182a-56de-4ab8-91de-3be01bf8fdc2\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") "
Sep 11 18:47:50.198764 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.198738    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-home\") pod \"8437182a-56de-4ab8-91de-3be01bf8fdc2\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") "
Sep 11 18:47:50.198764 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.198757    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-model-cache\") pod \"8437182a-56de-4ab8-91de-3be01bf8fdc2\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") "
Sep 11 18:47:50.198922 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.198775    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8437182a-56de-4ab8-91de-3be01bf8fdc2-tls-certs\") pod \"8437182a-56de-4ab8-91de-3be01bf8fdc2\" (UID: \"8437182a-56de-4ab8-91de-3be01bf8fdc2\") "
Sep 11 18:47:50.199102 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.198986    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-home" (OuterVolumeSpecName: "home") pod "8437182a-56de-4ab8-91de-3be01bf8fdc2" (UID: "8437182a-56de-4ab8-91de-3be01bf8fdc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:47:50.199102 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.199004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8437182a-56de-4ab8-91de-3be01bf8fdc2" (UID: "8437182a-56de-4ab8-91de-3be01bf8fdc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:47:50.199102 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.199036    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "8437182a-56de-4ab8-91de-3be01bf8fdc2" (UID: "8437182a-56de-4ab8-91de-3be01bf8fdc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:47:50.200804 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.200785    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8437182a-56de-4ab8-91de-3be01bf8fdc2-kube-api-access-6d26n" (OuterVolumeSpecName: "kube-api-access-6d26n") pod "8437182a-56de-4ab8-91de-3be01bf8fdc2" (UID: "8437182a-56de-4ab8-91de-3be01bf8fdc2"). InnerVolumeSpecName "kube-api-access-6d26n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:47:50.201265 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.201246    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8437182a-56de-4ab8-91de-3be01bf8fdc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8437182a-56de-4ab8-91de-3be01bf8fdc2" (UID: "8437182a-56de-4ab8-91de-3be01bf8fdc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:47:50.201265 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.201255    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-dshm" (OuterVolumeSpecName: "dshm") pod "8437182a-56de-4ab8-91de-3be01bf8fdc2" (UID: "8437182a-56de-4ab8-91de-3be01bf8fdc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:47:50.300365 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.300315    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:47:50.300365 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.300334    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8437182a-56de-4ab8-91de-3be01bf8fdc2-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:47:50.300365 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.300342    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d26n\" (UniqueName: \"kubernetes.io/projected/8437182a-56de-4ab8-91de-3be01bf8fdc2-kube-api-access-6d26n\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:47:50.300365 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.300351    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:47:50.300365 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.300360    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:47:50.300365 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.300367    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8437182a-56de-4ab8-91de-3be01bf8fdc2-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:47:50.347614 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.347590    2563 generic.go:356] "Generic (PLEG): container finished" podID="8437182a-56de-4ab8-91de-3be01bf8fdc2" containerID="bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4" exitCode=0
Sep 11 18:47:50.347718 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.347629    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"
Sep 11 18:47:50.347718 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.347682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh" event={"ID":"8437182a-56de-4ab8-91de-3be01bf8fdc2","Type":"ContainerDied","Data":"bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4"}
Sep 11 18:47:50.347797 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.347732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh" event={"ID":"8437182a-56de-4ab8-91de-3be01bf8fdc2","Type":"ContainerDied","Data":"29ae77923802cffda47738bdfbead0f09f4cebc127dc7a0567820d91cb8e88ce"}
Sep 11 18:47:50.347797 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.347751    2563 scope.go:117] "RemoveContainer" containerID="bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4"
Sep 11 18:47:50.356763 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.356746    2563 scope.go:117] "RemoveContainer" containerID="bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4"
Sep 11 18:47:50.357013 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:47:50.356996    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4\": container with ID starting with bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4 not found: ID does not exist" containerID="bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4"
Sep 11 18:47:50.357072 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.357025    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4"} err="failed to get container status \"bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4\": rpc error: code = NotFound desc = could not find container \"bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4\": container with ID starting with bce1187c131a8a2c557b1423df0ac083b1d08d0dd3ae99c484b749fb499444f4 not found: ID does not exist"
Sep 11 18:47:50.367989 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.367970    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"]
Sep 11 18:47:50.372294 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:50.372273    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/rate-test-model-ec9107-kserve-87949794c-b57wh"]
Sep 11 18:47:51.112779 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:51.112751    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8437182a-56de-4ab8-91de-3be01bf8fdc2" path="/var/lib/kubelet/pods/8437182a-56de-4ab8-91de-3be01bf8fdc2/volumes"
Sep 11 18:47:53.743028 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:47:53.742992    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:48:11.227276 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.227240    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"]
Sep 11 18:48:11.227747 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.227596    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32de7bf-cd55-47fb-9938-6471c3958cb5" containerName="main"
Sep 11 18:48:11.227747 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.227608    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32de7bf-cd55-47fb-9938-6471c3958cb5" containerName="main"
Sep 11 18:48:11.227747 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.227629    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8437182a-56de-4ab8-91de-3be01bf8fdc2" containerName="main"
Sep 11 18:48:11.227747 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.227635    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8437182a-56de-4ab8-91de-3be01bf8fdc2" containerName="main"
Sep 11 18:48:11.227747 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.227695    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8437182a-56de-4ab8-91de-3be01bf8fdc2" containerName="main"
Sep 11 18:48:11.227747 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.227707    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c32de7bf-cd55-47fb-9938-6471c3958cb5" containerName="main"
Sep 11 18:48:11.232051 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.232034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.234506 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.234484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b01b33-serving-cert\""
Sep 11 18:48:11.241402 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.241366    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"]
Sep 11 18:48:11.259357 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.259335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.259509 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.259399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.259509 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.259432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7m4s\" (UniqueName: \"kubernetes.io/projected/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-kube-api-access-b7m4s\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.259509 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.259472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-metrics-certs\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.360291 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.360264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.360431 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.360296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.360431 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.360315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7m4s\" (UniqueName: \"kubernetes.io/projected/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-kube-api-access-b7m4s\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.360431 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.360341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-metrics-certs\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.360570 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:48:11.360430    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-serving-cert: secret "maas-api-e2e-ipp-b-b01b33-serving-cert" not found
Sep 11 18:48:11.360570 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:48:11.360509    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-maas-api-tls podName:aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:11.860490116 +0000 UTC m=+1947.255005136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-maas-api-tls") pod "maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr" (UID: "aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84") : secret "maas-api-e2e-ipp-b-b01b33-serving-cert" not found
Sep 11 18:48:11.361065 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.361041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.362721 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.362703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-metrics-certs\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.368649 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.368631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7m4s\" (UniqueName: \"kubernetes.io/projected/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-kube-api-access-b7m4s\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.864230 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.864196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:11.866899 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:11.866870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:12.143776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:12.143688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:12.438434 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:12.438404    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"]
Sep 11 18:48:12.440289 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:48:12.440253    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4103f9_f7a6_4aa9_a30b_fc8c46c6ba84.slice/crio-00f5c48acf9164a1931a73778296c0704277e728880c79d8a227b99afbf0f55d WatchSource:0}: Error finding container 00f5c48acf9164a1931a73778296c0704277e728880c79d8a227b99afbf0f55d: Status 404 returned error can't find the container with id 00f5c48acf9164a1931a73778296c0704277e728880c79d8a227b99afbf0f55d
Sep 11 18:48:12.441717 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:12.441699    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:48:12.456099 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:12.456077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr" event={"ID":"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84","Type":"ContainerStarted","Data":"00f5c48acf9164a1931a73778296c0704277e728880c79d8a227b99afbf0f55d"}
Sep 11 18:48:13.463445 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:13.463406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr" event={"ID":"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84","Type":"ContainerStarted","Data":"9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64"}
Sep 11 18:48:13.463831 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:13.463480    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:48:13.482860 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:13.482821    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr" podStartSLOduration=2.123700453 podStartE2EDuration="2.482805097s" totalImagesPullingTime="359.104644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:13.482017221 +0000 UTC m=+1948.876532265" watchObservedRunningTime="2026-09-11 18:48:13.482805097 +0000 UTC m=+1948.877320122"
Sep 11 18:48:16.643556 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:16.643516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:48:19.481689 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:48:19.481663    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:49:00.455431 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.455395    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"]
Sep 11 18:49:00.458676 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.458660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.462352 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.462321    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-461acb\"/\"default-dockercfg-hcb4k\""
Sep 11 18:49:00.462519 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.462353    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-461acb\"/\"kube-root-ca.crt\""
Sep 11 18:49:00.462519 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.462330    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-461acb\"/\"ipp-route-461acb-kserve-self-signed-certs\""
Sep 11 18:49:00.462519 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.462323    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-461acb\"/\"openshift-service-ca.crt\""
Sep 11 18:49:00.468352 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.468331    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"]
Sep 11 18:49:00.532932 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.532905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-tmp-dir\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.533086 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.532935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm44d\" (UniqueName: \"kubernetes.io/projected/66bdcd52-6850-470d-a330-eb9bc88312ec-kube-api-access-tm44d\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.533086 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.532961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-model-cache\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.533086 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.532989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-dshm\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.533263 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.533113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66bdcd52-6850-470d-a330-eb9bc88312ec-tls-certs\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.533263 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.533175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-home\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.634287 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.634257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66bdcd52-6850-470d-a330-eb9bc88312ec-tls-certs\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.634453 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.634299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-home\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.634453 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.634323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-tmp-dir\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.634453 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.634340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm44d\" (UniqueName: \"kubernetes.io/projected/66bdcd52-6850-470d-a330-eb9bc88312ec-kube-api-access-tm44d\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.634453 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.634367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-model-cache\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.634453 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.634420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-dshm\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.634779 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.634697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-home\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.634779 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.634760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-tmp-dir\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.634870 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.634822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-model-cache\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.636727 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.636706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-dshm\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.636917 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.636898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66bdcd52-6850-470d-a330-eb9bc88312ec-tls-certs\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.643298 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.643279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm44d\" (UniqueName: \"kubernetes.io/projected/66bdcd52-6850-470d-a330-eb9bc88312ec-kube-api-access-tm44d\") pod \"ipp-route-461acb-kserve-6c4c95f44c-282nk\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") " pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:00.768549 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:00.768498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:01.038715 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:01.038692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"]
Sep 11 18:49:01.040560 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:49:01.040529    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66bdcd52_6850_470d_a330_eb9bc88312ec.slice/crio-001659f8cc453c00bdd4360dbc2031aceed2be8f8f378a41266c6706c4889ce8 WatchSource:0}: Error finding container 001659f8cc453c00bdd4360dbc2031aceed2be8f8f378a41266c6706c4889ce8: Status 404 returned error can't find the container with id 001659f8cc453c00bdd4360dbc2031aceed2be8f8f378a41266c6706c4889ce8
Sep 11 18:49:01.717636 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:01.717604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk" event={"ID":"66bdcd52-6850-470d-a330-eb9bc88312ec","Type":"ContainerStarted","Data":"5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85"}
Sep 11 18:49:01.718036 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:01.717649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk" event={"ID":"66bdcd52-6850-470d-a330-eb9bc88312ec","Type":"ContainerStarted","Data":"001659f8cc453c00bdd4360dbc2031aceed2be8f8f378a41266c6706c4889ce8"}
Sep 11 18:49:01.736734 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:01.736688    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk" podStartSLOduration=1.736669557 podStartE2EDuration="1.736669557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:01.734346243 +0000 UTC m=+1997.128861290" watchObservedRunningTime="2026-09-11 18:49:01.736669557 +0000 UTC m=+1997.131184583"
Sep 11 18:49:04.646880 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:04.646847    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:49:10.769156 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:10.769122    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:10.769608 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:10.769168    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:10.781596 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:10.781571    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:11.777780 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:11.777751    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:49:17.850244 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:17.850212    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:49:31.529087 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.529048    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"]
Sep 11 18:49:31.532698 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.532682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:49:31.535472 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.535432    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d12501-dockercfg-q7bf7\""
Sep 11 18:49:31.535591 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.535569    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d12501\""
Sep 11 18:49:31.544134 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.544109    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"]
Sep 11 18:49:31.565973 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.565950    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"]
Sep 11 18:49:31.569347 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.569330    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:49:31.574023 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.573989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n\" (UID: \"69fadb8d-b350-469a-81a6-4d88dd4f8e6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:49:31.574251 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.574221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptr4s\" (UniqueName: \"kubernetes.io/projected/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-kube-api-access-ptr4s\") pod \"payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n\" (UID: \"69fadb8d-b350-469a-81a6-4d88dd4f8e6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:49:31.579563 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.579541    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"]
Sep 11 18:49:31.675444 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.675411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01d66d1-afda-4efd-912b-063a27aeea72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c\" (UID: \"a01d66d1-afda-4efd-912b-063a27aeea72\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:49:31.675545 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.675476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptr4s\" (UniqueName: \"kubernetes.io/projected/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-kube-api-access-ptr4s\") pod \"payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n\" (UID: \"69fadb8d-b350-469a-81a6-4d88dd4f8e6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:49:31.675545 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.675501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgslh\" (UniqueName: \"kubernetes.io/projected/a01d66d1-afda-4efd-912b-063a27aeea72-kube-api-access-cgslh\") pod \"payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c\" (UID: \"a01d66d1-afda-4efd-912b-063a27aeea72\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:49:31.675630 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.675577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n\" (UID: \"69fadb8d-b350-469a-81a6-4d88dd4f8e6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:49:31.676231 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.676213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n\" (UID: \"69fadb8d-b350-469a-81a6-4d88dd4f8e6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:49:31.684044 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.684021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptr4s\" (UniqueName: \"kubernetes.io/projected/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-kube-api-access-ptr4s\") pod \"payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n\" (UID: \"69fadb8d-b350-469a-81a6-4d88dd4f8e6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:49:31.776276 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.776248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01d66d1-afda-4efd-912b-063a27aeea72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c\" (UID: \"a01d66d1-afda-4efd-912b-063a27aeea72\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:49:31.776422 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.776305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgslh\" (UniqueName: \"kubernetes.io/projected/a01d66d1-afda-4efd-912b-063a27aeea72-kube-api-access-cgslh\") pod \"payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c\" (UID: \"a01d66d1-afda-4efd-912b-063a27aeea72\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:49:31.776876 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.776858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01d66d1-afda-4efd-912b-063a27aeea72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c\" (UID: \"a01d66d1-afda-4efd-912b-063a27aeea72\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:49:31.787441 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.787395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgslh\" (UniqueName: \"kubernetes.io/projected/a01d66d1-afda-4efd-912b-063a27aeea72-kube-api-access-cgslh\") pod \"payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c\" (UID: \"a01d66d1-afda-4efd-912b-063a27aeea72\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:49:31.844591 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.844559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:49:31.881133 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:31.881104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:49:32.119358 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:32.119332    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"]
Sep 11 18:49:32.121134 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:49:32.121105    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fadb8d_b350_469a_81a6_4d88dd4f8e6e.slice/crio-de09467adb6a58c49b8cec5faf59b9dc12a256326f26dd1fcb43660307d264f9 WatchSource:0}: Error finding container de09467adb6a58c49b8cec5faf59b9dc12a256326f26dd1fcb43660307d264f9: Status 404 returned error can't find the container with id de09467adb6a58c49b8cec5faf59b9dc12a256326f26dd1fcb43660307d264f9
Sep 11 18:49:32.171649 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:32.171630    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"]
Sep 11 18:49:32.173135 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:49:32.173111    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01d66d1_afda_4efd_912b_063a27aeea72.slice/crio-a8ef09773caf9ab3de2304463ca8fad85e3530c7a3018813a098a7e5186c04a8 WatchSource:0}: Error finding container a8ef09773caf9ab3de2304463ca8fad85e3530c7a3018813a098a7e5186c04a8: Status 404 returned error can't find the container with id a8ef09773caf9ab3de2304463ca8fad85e3530c7a3018813a098a7e5186c04a8
Sep 11 18:49:32.870421 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:32.870356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c" event={"ID":"a01d66d1-afda-4efd-912b-063a27aeea72","Type":"ContainerStarted","Data":"6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d"}
Sep 11 18:49:32.870870 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:32.870430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c" event={"ID":"a01d66d1-afda-4efd-912b-063a27aeea72","Type":"ContainerStarted","Data":"a8ef09773caf9ab3de2304463ca8fad85e3530c7a3018813a098a7e5186c04a8"}
Sep 11 18:49:32.870870 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:32.870472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:49:32.872061 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:32.872036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n" event={"ID":"69fadb8d-b350-469a-81a6-4d88dd4f8e6e","Type":"ContainerStarted","Data":"2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f"}
Sep 11 18:49:32.872184 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:32.872084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n" event={"ID":"69fadb8d-b350-469a-81a6-4d88dd4f8e6e","Type":"ContainerStarted","Data":"de09467adb6a58c49b8cec5faf59b9dc12a256326f26dd1fcb43660307d264f9"}
Sep 11 18:49:32.872184 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:32.872127    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:49:32.891575 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:32.891520    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c" podStartSLOduration=1.891502505 podStartE2EDuration="1.891502505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:32.888261866 +0000 UTC m=+2028.282776923" watchObservedRunningTime="2026-09-11 18:49:32.891502505 +0000 UTC m=+2028.286017530"
Sep 11 18:49:32.904803 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:32.904758    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n" podStartSLOduration=1.904746514 podStartE2EDuration="1.904746514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:32.903292694 +0000 UTC m=+2028.297807748" watchObservedRunningTime="2026-09-11 18:49:32.904746514 +0000 UTC m=+2028.299261542"
Sep 11 18:49:37.319304 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:37.319264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:49:43.880824 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:43.880795    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:49:43.881207 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:43.880962    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:49:45.354676 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:45.354645    2563 scope.go:117] "RemoveContainer" containerID="93a6a127b990db1a5ce89514d9c8a9825691a74099d2bd24b613c697f7435799"
Sep 11 18:49:59.900285 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:59.900007    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"]
Sep 11 18:49:59.900285 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:59.900241    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c" podUID="a01d66d1-afda-4efd-912b-063a27aeea72" containerName="payload-processing" containerID="cri-o://6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d" gracePeriod=30
Sep 11 18:49:59.906532 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:59.906487    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"]
Sep 11 18:49:59.906791 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:49:59.906753    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n" podUID="69fadb8d-b350-469a-81a6-4d88dd4f8e6e" containerName="payload-pre-processing" containerID="cri-o://2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f" gracePeriod=30
Sep 11 18:50:00.180894 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.180868    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:50:00.183625 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.183607    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:50:00.315603 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.315572    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptr4s\" (UniqueName: \"kubernetes.io/projected/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-kube-api-access-ptr4s\") pod \"69fadb8d-b350-469a-81a6-4d88dd4f8e6e\" (UID: \"69fadb8d-b350-469a-81a6-4d88dd4f8e6e\") "
Sep 11 18:50:00.315758 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.315611    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01d66d1-afda-4efd-912b-063a27aeea72-plugins-config-volume\") pod \"a01d66d1-afda-4efd-912b-063a27aeea72\" (UID: \"a01d66d1-afda-4efd-912b-063a27aeea72\") "
Sep 11 18:50:00.315758 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.315695    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-plugins-config-volume\") pod \"69fadb8d-b350-469a-81a6-4d88dd4f8e6e\" (UID: \"69fadb8d-b350-469a-81a6-4d88dd4f8e6e\") "
Sep 11 18:50:00.315885 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.315803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgslh\" (UniqueName: \"kubernetes.io/projected/a01d66d1-afda-4efd-912b-063a27aeea72-kube-api-access-cgslh\") pod \"a01d66d1-afda-4efd-912b-063a27aeea72\" (UID: \"a01d66d1-afda-4efd-912b-063a27aeea72\") "
Sep 11 18:50:00.316072 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.316022    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69fadb8d-b350-469a-81a6-4d88dd4f8e6e" (UID: "69fadb8d-b350-469a-81a6-4d88dd4f8e6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:50:00.316072 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.316043    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01d66d1-afda-4efd-912b-063a27aeea72-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a01d66d1-afda-4efd-912b-063a27aeea72" (UID: "a01d66d1-afda-4efd-912b-063a27aeea72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:50:00.316228 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.316175    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01d66d1-afda-4efd-912b-063a27aeea72-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.316228 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.316196    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.317774 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.317753    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01d66d1-afda-4efd-912b-063a27aeea72-kube-api-access-cgslh" (OuterVolumeSpecName: "kube-api-access-cgslh") pod "a01d66d1-afda-4efd-912b-063a27aeea72" (UID: "a01d66d1-afda-4efd-912b-063a27aeea72"). InnerVolumeSpecName "kube-api-access-cgslh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:00.317850 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.317771    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-kube-api-access-ptr4s" (OuterVolumeSpecName: "kube-api-access-ptr4s") pod "69fadb8d-b350-469a-81a6-4d88dd4f8e6e" (UID: "69fadb8d-b350-469a-81a6-4d88dd4f8e6e"). InnerVolumeSpecName "kube-api-access-ptr4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:00.416755 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.416734    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgslh\" (UniqueName: \"kubernetes.io/projected/a01d66d1-afda-4efd-912b-063a27aeea72-kube-api-access-cgslh\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.416755 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:00.416754    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptr4s\" (UniqueName: \"kubernetes.io/projected/69fadb8d-b350-469a-81a6-4d88dd4f8e6e-kube-api-access-ptr4s\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:50:01.014996 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.014965    2563 generic.go:356] "Generic (PLEG): container finished" podID="69fadb8d-b350-469a-81a6-4d88dd4f8e6e" containerID="2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f" exitCode=0
Sep 11 18:50:01.015425 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.015003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n" event={"ID":"69fadb8d-b350-469a-81a6-4d88dd4f8e6e","Type":"ContainerDied","Data":"2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f"}
Sep 11 18:50:01.015425 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.015076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n" event={"ID":"69fadb8d-b350-469a-81a6-4d88dd4f8e6e","Type":"ContainerDied","Data":"de09467adb6a58c49b8cec5faf59b9dc12a256326f26dd1fcb43660307d264f9"}
Sep 11 18:50:01.015425 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.015014    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"
Sep 11 18:50:01.015425 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.015113    2563 scope.go:117] "RemoveContainer" containerID="2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f"
Sep 11 18:50:01.016571 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.016544    2563 generic.go:356] "Generic (PLEG): container finished" podID="a01d66d1-afda-4efd-912b-063a27aeea72" containerID="6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d" exitCode=0
Sep 11 18:50:01.016638 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.016573    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c" event={"ID":"a01d66d1-afda-4efd-912b-063a27aeea72","Type":"ContainerDied","Data":"6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d"}
Sep 11 18:50:01.016638 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.016601    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"
Sep 11 18:50:01.016638 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.016613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c" event={"ID":"a01d66d1-afda-4efd-912b-063a27aeea72","Type":"ContainerDied","Data":"a8ef09773caf9ab3de2304463ca8fad85e3530c7a3018813a098a7e5186c04a8"}
Sep 11 18:50:01.023814 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.023796    2563 scope.go:117] "RemoveContainer" containerID="2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f"
Sep 11 18:50:01.024054 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:50:01.024031    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f\": container with ID starting with 2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f not found: ID does not exist" containerID="2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f"
Sep 11 18:50:01.024120 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.024060    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f"} err="failed to get container status \"2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f\": rpc error: code = NotFound desc = could not find container \"2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f\": container with ID starting with 2a7c2c0afe7d6199b2893c7b5c22f3f8d610ddf5a0439f652addfac2da42dc4f not found: ID does not exist"
Sep 11 18:50:01.024120 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.024080    2563 scope.go:117] "RemoveContainer" containerID="6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d"
Sep 11 18:50:01.032224 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.032206    2563 scope.go:117] "RemoveContainer" containerID="6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d"
Sep 11 18:50:01.032555 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:50:01.032536    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d\": container with ID starting with 6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d not found: ID does not exist" containerID="6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d"
Sep 11 18:50:01.032621 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.032560    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d"} err="failed to get container status \"6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d\": rpc error: code = NotFound desc = could not find container \"6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d\": container with ID starting with 6ea509ee6c3e67c2172d20ebcdf05179c7814c717ab6cb341edf985434d4427d not found: ID does not exist"
Sep 11 18:50:01.041692 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.041670    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"]
Sep 11 18:50:01.045450 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.045432    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d12501-7776c98ffd-xhp4n"]
Sep 11 18:50:01.056350 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.056326    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"]
Sep 11 18:50:01.063601 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.063579    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d12501-57748bbc7c-df92c"]
Sep 11 18:50:01.112565 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.112546    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69fadb8d-b350-469a-81a6-4d88dd4f8e6e" path="/var/lib/kubelet/pods/69fadb8d-b350-469a-81a6-4d88dd4f8e6e/volumes"
Sep 11 18:50:01.112950 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:01.112936    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a01d66d1-afda-4efd-912b-063a27aeea72" path="/var/lib/kubelet/pods/a01d66d1-afda-4efd-912b-063a27aeea72/volumes"
Sep 11 18:50:04.048475 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:04.048443    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:50:23.648711 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:23.648660    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:50:31.256557 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:31.256517    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"]
Sep 11 18:50:35.455907 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:35.455875    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:50:45.217506 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:45.217476    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:50:45.224160 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:45.224129    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:50:45.225173 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:45.225149    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:50:45.231958 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:45.231940    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:50:45.373447 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:45.373424    2563 scope.go:117] "RemoveContainer" containerID="5bf34a74dac012ded014c4db741b08d42fa91ae67614d33b754d73cb09b2e264"
Sep 11 18:50:46.278337 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.278288    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk" podUID="66bdcd52-6850-470d-a330-eb9bc88312ec" containerName="main" containerID="cri-o://5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85" gracePeriod=45
Sep 11 18:50:46.518671 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.518651    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:50:46.562670 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.562600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-tmp-dir\") pod \"66bdcd52-6850-470d-a330-eb9bc88312ec\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") "
Sep 11 18:50:46.562670 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.562656    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66bdcd52-6850-470d-a330-eb9bc88312ec-tls-certs\") pod \"66bdcd52-6850-470d-a330-eb9bc88312ec\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") "
Sep 11 18:50:46.562867 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.562713    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-model-cache\") pod \"66bdcd52-6850-470d-a330-eb9bc88312ec\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") "
Sep 11 18:50:46.562867 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.562746    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm44d\" (UniqueName: \"kubernetes.io/projected/66bdcd52-6850-470d-a330-eb9bc88312ec-kube-api-access-tm44d\") pod \"66bdcd52-6850-470d-a330-eb9bc88312ec\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") "
Sep 11 18:50:46.562867 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.562811    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-dshm\") pod \"66bdcd52-6850-470d-a330-eb9bc88312ec\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") "
Sep 11 18:50:46.562867 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.562865    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-home\") pod \"66bdcd52-6850-470d-a330-eb9bc88312ec\" (UID: \"66bdcd52-6850-470d-a330-eb9bc88312ec\") "
Sep 11 18:50:46.563073 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.562905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66bdcd52-6850-470d-a330-eb9bc88312ec" (UID: "66bdcd52-6850-470d-a330-eb9bc88312ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:46.563073 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.563060    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "66bdcd52-6850-470d-a330-eb9bc88312ec" (UID: "66bdcd52-6850-470d-a330-eb9bc88312ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:46.563285 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.563265    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:50:46.563285 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.563274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-home" (OuterVolumeSpecName: "home") pod "66bdcd52-6850-470d-a330-eb9bc88312ec" (UID: "66bdcd52-6850-470d-a330-eb9bc88312ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:46.563431 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.563291    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:50:46.565049 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.565016    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66bdcd52-6850-470d-a330-eb9bc88312ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66bdcd52-6850-470d-a330-eb9bc88312ec" (UID: "66bdcd52-6850-470d-a330-eb9bc88312ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:50:46.565172 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.565123    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bdcd52-6850-470d-a330-eb9bc88312ec-kube-api-access-tm44d" (OuterVolumeSpecName: "kube-api-access-tm44d") pod "66bdcd52-6850-470d-a330-eb9bc88312ec" (UID: "66bdcd52-6850-470d-a330-eb9bc88312ec"). InnerVolumeSpecName "kube-api-access-tm44d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:46.565235 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.565168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-dshm" (OuterVolumeSpecName: "dshm") pod "66bdcd52-6850-470d-a330-eb9bc88312ec" (UID: "66bdcd52-6850-470d-a330-eb9bc88312ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:46.664243 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.664220    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:50:46.664331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.664242    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66bdcd52-6850-470d-a330-eb9bc88312ec-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:50:46.664331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.664268    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm44d\" (UniqueName: \"kubernetes.io/projected/66bdcd52-6850-470d-a330-eb9bc88312ec-kube-api-access-tm44d\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:50:46.664331 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:46.664277    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66bdcd52-6850-470d-a330-eb9bc88312ec-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:50:47.240582 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:47.240556    2563 generic.go:356] "Generic (PLEG): container finished" podID="66bdcd52-6850-470d-a330-eb9bc88312ec" containerID="5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85" exitCode=0
Sep 11 18:50:47.240690 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:47.240590    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"
Sep 11 18:50:47.240690 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:47.240643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk" event={"ID":"66bdcd52-6850-470d-a330-eb9bc88312ec","Type":"ContainerDied","Data":"5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85"}
Sep 11 18:50:47.240690 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:47.240688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk" event={"ID":"66bdcd52-6850-470d-a330-eb9bc88312ec","Type":"ContainerDied","Data":"001659f8cc453c00bdd4360dbc2031aceed2be8f8f378a41266c6706c4889ce8"}
Sep 11 18:50:47.240915 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:47.240710    2563 scope.go:117] "RemoveContainer" containerID="5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85"
Sep 11 18:50:47.249171 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:47.249153    2563 scope.go:117] "RemoveContainer" containerID="5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85"
Sep 11 18:50:47.249467 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:50:47.249438    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85\": container with ID starting with 5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85 not found: ID does not exist" containerID="5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85"
Sep 11 18:50:47.249537 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:47.249477    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85"} err="failed to get container status \"5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85\": rpc error: code = NotFound desc = could not find container \"5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85\": container with ID starting with 5197c49bb15ebe834132b07c8e61f11c6b9d866f5e5421a2ec55167b5f81db85 not found: ID does not exist"
Sep 11 18:50:47.257679 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:47.257658    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"]
Sep 11 18:50:47.261755 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:47.261735    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-461acb/ipp-route-461acb-kserve-6c4c95f44c-282nk"]
Sep 11 18:50:49.114857 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:50:49.114815    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66bdcd52-6850-470d-a330-eb9bc88312ec" path="/var/lib/kubelet/pods/66bdcd52-6850-470d-a330-eb9bc88312ec/volumes"
Sep 11 18:51:25.479996 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.479962    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"]
Sep 11 18:51:25.481153 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.481121    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr" podUID="aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84" containerName="maas-api" containerID="cri-o://9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64" gracePeriod=30
Sep 11 18:51:25.735989 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.735939    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:51:25.849927 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.849901    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7m4s\" (UniqueName: \"kubernetes.io/projected/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-kube-api-access-b7m4s\") pod \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") "
Sep 11 18:51:25.850077 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.849945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-metrics-certs\") pod \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") "
Sep 11 18:51:25.850077 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.850020    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-maas-api-tls\") pod \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") "
Sep 11 18:51:25.850189 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.850077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-openshift-service-ca\") pod \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\" (UID: \"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84\") "
Sep 11 18:51:25.850732 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.850705    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84" (UID: "aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:51:25.852492 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.852455    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84" (UID: "aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:51:25.852612 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.852596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84" (UID: "aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:51:25.852677 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.852630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-kube-api-access-b7m4s" (OuterVolumeSpecName: "kube-api-access-b7m4s") pod "aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84" (UID: "aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84"). InnerVolumeSpecName "kube-api-access-b7m4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:51:25.950911 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.950884    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-openshift-service-ca\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:51:25.950911 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.950908    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7m4s\" (UniqueName: \"kubernetes.io/projected/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-kube-api-access-b7m4s\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:51:25.951039 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.950919    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-metrics-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:51:25.951039 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:25.950929    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84-maas-api-tls\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:51:26.438686 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:26.438658    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84" containerID="9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64" exitCode=0
Sep 11 18:51:26.438833 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:26.438706    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"
Sep 11 18:51:26.438833 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:26.438716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr" event={"ID":"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84","Type":"ContainerDied","Data":"9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64"}
Sep 11 18:51:26.438833 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:26.438756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr" event={"ID":"aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84","Type":"ContainerDied","Data":"00f5c48acf9164a1931a73778296c0704277e728880c79d8a227b99afbf0f55d"}
Sep 11 18:51:26.438833 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:26.438784    2563 scope.go:117] "RemoveContainer" containerID="9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64"
Sep 11 18:51:26.447662 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:26.447641    2563 scope.go:117] "RemoveContainer" containerID="9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64"
Sep 11 18:51:26.447945 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:51:26.447926    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64\": container with ID starting with 9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64 not found: ID does not exist" containerID="9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64"
Sep 11 18:51:26.447985 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:26.447953    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64"} err="failed to get container status \"9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64\": rpc error: code = NotFound desc = could not find container \"9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64\": container with ID starting with 9387364f3fb71d50cd59c45c92873f1ef41c46311ae600f9b3fcc16362f4bb64 not found: ID does not exist"
Sep 11 18:51:26.461238 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:26.461213    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"]
Sep 11 18:51:26.465467 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:26.465449    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b01b33-6ff8dd8fc-9m4rr"]
Sep 11 18:51:27.113755 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:27.113725    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84" path="/var/lib/kubelet/pods/aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84/volumes"
Sep 11 18:51:29.004835 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:29.004803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:51:42.895816 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:42.895718    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"]
Sep 11 18:51:42.896250 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:42.895950    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc" podUID="464ca645-1810-49d0-a50f-1435c35644f2" containerName="maas-api" containerID="cri-o://642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee" gracePeriod=30
Sep 11 18:51:43.148157 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.148105    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:51:43.283976 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.283952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464ca645-1810-49d0-a50f-1435c35644f2-openshift-service-ca\") pod \"464ca645-1810-49d0-a50f-1435c35644f2\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") "
Sep 11 18:51:43.284092 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.284007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-maas-api-tls\") pod \"464ca645-1810-49d0-a50f-1435c35644f2\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") "
Sep 11 18:51:43.284092 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.284031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7ztq\" (UniqueName: \"kubernetes.io/projected/464ca645-1810-49d0-a50f-1435c35644f2-kube-api-access-j7ztq\") pod \"464ca645-1810-49d0-a50f-1435c35644f2\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") "
Sep 11 18:51:43.284092 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.284069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-metrics-certs\") pod \"464ca645-1810-49d0-a50f-1435c35644f2\" (UID: \"464ca645-1810-49d0-a50f-1435c35644f2\") "
Sep 11 18:51:43.284314 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.284291    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464ca645-1810-49d0-a50f-1435c35644f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "464ca645-1810-49d0-a50f-1435c35644f2" (UID: "464ca645-1810-49d0-a50f-1435c35644f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:51:43.286063 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.286036    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "464ca645-1810-49d0-a50f-1435c35644f2" (UID: "464ca645-1810-49d0-a50f-1435c35644f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:51:43.286148 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.286078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "464ca645-1810-49d0-a50f-1435c35644f2" (UID: "464ca645-1810-49d0-a50f-1435c35644f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:51:43.286148 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.286088    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464ca645-1810-49d0-a50f-1435c35644f2-kube-api-access-j7ztq" (OuterVolumeSpecName: "kube-api-access-j7ztq") pod "464ca645-1810-49d0-a50f-1435c35644f2" (UID: "464ca645-1810-49d0-a50f-1435c35644f2"). InnerVolumeSpecName "kube-api-access-j7ztq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:51:43.384955 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.384929    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-maas-api-tls\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:51:43.384955 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.384954    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7ztq\" (UniqueName: \"kubernetes.io/projected/464ca645-1810-49d0-a50f-1435c35644f2-kube-api-access-j7ztq\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:51:43.385084 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.384965    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464ca645-1810-49d0-a50f-1435c35644f2-metrics-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:51:43.385084 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.384972    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464ca645-1810-49d0-a50f-1435c35644f2-openshift-service-ca\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:51:43.533784 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.533754    2563 generic.go:356] "Generic (PLEG): container finished" podID="464ca645-1810-49d0-a50f-1435c35644f2" containerID="642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee" exitCode=0
Sep 11 18:51:43.533925 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.533831    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"
Sep 11 18:51:43.533925 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.533837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc" event={"ID":"464ca645-1810-49d0-a50f-1435c35644f2","Type":"ContainerDied","Data":"642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee"}
Sep 11 18:51:43.534047 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.533927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc" event={"ID":"464ca645-1810-49d0-a50f-1435c35644f2","Type":"ContainerDied","Data":"d1fbf183e51f78023cbb000226ea48b0ff84a4e321f987d4ad78ab762512c50e"}
Sep 11 18:51:43.534047 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.533958    2563 scope.go:117] "RemoveContainer" containerID="642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee"
Sep 11 18:51:43.542977 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.542961    2563 scope.go:117] "RemoveContainer" containerID="642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee"
Sep 11 18:51:43.543227 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:51:43.543210    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee\": container with ID starting with 642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee not found: ID does not exist" containerID="642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee"
Sep 11 18:51:43.543274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.543233    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee"} err="failed to get container status \"642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee\": rpc error: code = NotFound desc = could not find container \"642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee\": container with ID starting with 642c4c0fb4307a092afe12d1b43378d0be56c46ebf32064a079c95e0f2535bee not found: ID does not exist"
Sep 11 18:51:43.555885 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.555861    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"]
Sep 11 18:51:43.563381 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:43.563348    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-ec9107-8687556bd-6kmsc"]
Sep 11 18:51:45.112827 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:45.112799    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464ca645-1810-49d0-a50f-1435c35644f2" path="/var/lib/kubelet/pods/464ca645-1810-49d0-a50f-1435c35644f2/volumes"
Sep 11 18:51:45.760178 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:45.760151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:51:50.555362 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555327    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"]
Sep 11 18:51:50.555736 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555721    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69fadb8d-b350-469a-81a6-4d88dd4f8e6e" containerName="payload-pre-processing"
Sep 11 18:51:50.555802 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555737    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69fadb8d-b350-469a-81a6-4d88dd4f8e6e" containerName="payload-pre-processing"
Sep 11 18:51:50.555802 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555754    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bdcd52-6850-470d-a330-eb9bc88312ec" containerName="main"
Sep 11 18:51:50.555802 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555760    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bdcd52-6850-470d-a330-eb9bc88312ec" containerName="main"
Sep 11 18:51:50.555802 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555769    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01d66d1-afda-4efd-912b-063a27aeea72" containerName="payload-processing"
Sep 11 18:51:50.555802 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555775    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01d66d1-afda-4efd-912b-063a27aeea72" containerName="payload-processing"
Sep 11 18:51:50.555802 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555800    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464ca645-1810-49d0-a50f-1435c35644f2" containerName="maas-api"
Sep 11 18:51:50.555983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555807    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="464ca645-1810-49d0-a50f-1435c35644f2" containerName="maas-api"
Sep 11 18:51:50.555983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555815    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84" containerName="maas-api"
Sep 11 18:51:50.555983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555820    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84" containerName="maas-api"
Sep 11 18:51:50.555983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555874    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="464ca645-1810-49d0-a50f-1435c35644f2" containerName="maas-api"
Sep 11 18:51:50.555983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555882    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="66bdcd52-6850-470d-a330-eb9bc88312ec" containerName="main"
Sep 11 18:51:50.555983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555889    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69fadb8d-b350-469a-81a6-4d88dd4f8e6e" containerName="payload-pre-processing"
Sep 11 18:51:50.555983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555895    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa4103f9-f7a6-4aa9-a30b-fc8c46c6ba84" containerName="maas-api"
Sep 11 18:51:50.555983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.555901    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a01d66d1-afda-4efd-912b-063a27aeea72" containerName="payload-processing"
Sep 11 18:51:50.559083 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.559061    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"
Sep 11 18:51:50.561766 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.561739    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:51:50.561875 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.561753    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:51:50.568823 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.568749    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"]
Sep 11 18:51:50.641069 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.641040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971e1052-d1d1-4f4b-a13e-a0419bc7f360-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9\" (UID: \"971e1052-d1d1-4f4b-a13e-a0419bc7f360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"
Sep 11 18:51:50.742274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.742247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971e1052-d1d1-4f4b-a13e-a0419bc7f360-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9\" (UID: \"971e1052-d1d1-4f4b-a13e-a0419bc7f360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"
Sep 11 18:51:50.742850 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.742833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971e1052-d1d1-4f4b-a13e-a0419bc7f360-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9\" (UID: \"971e1052-d1d1-4f4b-a13e-a0419bc7f360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"
Sep 11 18:51:50.873402 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:50.873334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"
Sep 11 18:51:51.140669 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:51.140643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"]
Sep 11 18:51:51.142336 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:51:51.142308    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971e1052_d1d1_4f4b_a13e_a0419bc7f360.slice/crio-dc0478f6abbbde6c140b44c8d46816d5fcea5c03dd9d0d50980cda6762e1b242 WatchSource:0}: Error finding container dc0478f6abbbde6c140b44c8d46816d5fcea5c03dd9d0d50980cda6762e1b242: Status 404 returned error can't find the container with id dc0478f6abbbde6c140b44c8d46816d5fcea5c03dd9d0d50980cda6762e1b242
Sep 11 18:51:51.577056 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:51.577023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9" event={"ID":"971e1052-d1d1-4f4b-a13e-a0419bc7f360","Type":"ContainerStarted","Data":"23ed19e9894e12a405f8a68f37ef38eae5949c857dba30d390843879c437c0fb"}
Sep 11 18:51:51.577342 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:51.577078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9" event={"ID":"971e1052-d1d1-4f4b-a13e-a0419bc7f360","Type":"ContainerStarted","Data":"dc0478f6abbbde6c140b44c8d46816d5fcea5c03dd9d0d50980cda6762e1b242"}
Sep 11 18:51:51.594175 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:51.594135    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9" podStartSLOduration=1.594118328 podStartE2EDuration="1.594118328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:51:51.593141288 +0000 UTC m=+2166.987656338" watchObservedRunningTime="2026-09-11 18:51:51.594118328 +0000 UTC m=+2166.988633354"
Sep 11 18:51:52.582887 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:52.582846    2563 generic.go:356] "Generic (PLEG): container finished" podID="971e1052-d1d1-4f4b-a13e-a0419bc7f360" containerID="23ed19e9894e12a405f8a68f37ef38eae5949c857dba30d390843879c437c0fb" exitCode=0
Sep 11 18:51:52.583245 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:52.582926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9" event={"ID":"971e1052-d1d1-4f4b-a13e-a0419bc7f360","Type":"ContainerDied","Data":"23ed19e9894e12a405f8a68f37ef38eae5949c857dba30d390843879c437c0fb"}
Sep 11 18:51:53.716411 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:53.716386    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"
Sep 11 18:51:53.868072 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:53.868016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971e1052-d1d1-4f4b-a13e-a0419bc7f360-maas-api-service-ca\") pod \"971e1052-d1d1-4f4b-a13e-a0419bc7f360\" (UID: \"971e1052-d1d1-4f4b-a13e-a0419bc7f360\") "
Sep 11 18:51:53.868338 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:53.868316    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971e1052-d1d1-4f4b-a13e-a0419bc7f360-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "971e1052-d1d1-4f4b-a13e-a0419bc7f360" (UID: "971e1052-d1d1-4f4b-a13e-a0419bc7f360"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:51:53.968923 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:53.968895    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971e1052-d1d1-4f4b-a13e-a0419bc7f360-maas-api-service-ca\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:51:54.593533 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:54.593506    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"
Sep 11 18:51:54.593669 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:54.593501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9" event={"ID":"971e1052-d1d1-4f4b-a13e-a0419bc7f360","Type":"ContainerDied","Data":"dc0478f6abbbde6c140b44c8d46816d5fcea5c03dd9d0d50980cda6762e1b242"}
Sep 11 18:51:54.593669 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:51:54.593654    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc0478f6abbbde6c140b44c8d46816d5fcea5c03dd9d0d50980cda6762e1b242"
Sep 11 18:52:01.034980 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.034941    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz"]
Sep 11 18:52:01.040150 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.040123    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d6c4cc-29819205-gzkrz"]
Sep 11 18:52:01.055514 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.055487    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"]
Sep 11 18:52:01.055706 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.055684    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz" podUID="8cce9724-afd3-4690-966a-1fd6f2cd0251" containerName="payload-processing" containerID="cri-o://7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe" gracePeriod=30
Sep 11 18:52:01.067864 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.067408    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"]
Sep 11 18:52:01.068251 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.068200    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5" podUID="92692b23-1a8a-4890-8af7-e9c8da0e17b5" containerName="payload-pre-processing" containerID="cri-o://868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e" gracePeriod=30
Sep 11 18:52:01.112843 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.112812    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1963cbca-2880-4a0c-926a-7f3da46e68b9" path="/var/lib/kubelet/pods/1963cbca-2880-4a0c-926a-7f3da46e68b9/volumes"
Sep 11 18:52:01.230710 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.229767    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"]
Sep 11 18:52:01.232709 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.232209    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6c4cc-dc8p9"]
Sep 11 18:52:01.310976 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.310956    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:52:01.339409 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.339384    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:52:01.422368 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.422342    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsx62\" (UniqueName: \"kubernetes.io/projected/8cce9724-afd3-4690-966a-1fd6f2cd0251-kube-api-access-wsx62\") pod \"8cce9724-afd3-4690-966a-1fd6f2cd0251\" (UID: \"8cce9724-afd3-4690-966a-1fd6f2cd0251\") "
Sep 11 18:52:01.422535 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.422433    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92692b23-1a8a-4890-8af7-e9c8da0e17b5-plugins-config-volume\") pod \"92692b23-1a8a-4890-8af7-e9c8da0e17b5\" (UID: \"92692b23-1a8a-4890-8af7-e9c8da0e17b5\") "
Sep 11 18:52:01.422602 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.422541    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrk2p\" (UniqueName: \"kubernetes.io/projected/92692b23-1a8a-4890-8af7-e9c8da0e17b5-kube-api-access-xrk2p\") pod \"92692b23-1a8a-4890-8af7-e9c8da0e17b5\" (UID: \"92692b23-1a8a-4890-8af7-e9c8da0e17b5\") "
Sep 11 18:52:01.422655 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.422612    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cce9724-afd3-4690-966a-1fd6f2cd0251-plugins-config-volume\") pod \"8cce9724-afd3-4690-966a-1fd6f2cd0251\" (UID: \"8cce9724-afd3-4690-966a-1fd6f2cd0251\") "
Sep 11 18:52:01.422800 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.422774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92692b23-1a8a-4890-8af7-e9c8da0e17b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "92692b23-1a8a-4890-8af7-e9c8da0e17b5" (UID: "92692b23-1a8a-4890-8af7-e9c8da0e17b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:52:01.422960 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.422944    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92692b23-1a8a-4890-8af7-e9c8da0e17b5-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:52:01.423019 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.422984    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cce9724-afd3-4690-966a-1fd6f2cd0251-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cce9724-afd3-4690-966a-1fd6f2cd0251" (UID: "8cce9724-afd3-4690-966a-1fd6f2cd0251"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:52:01.424448 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.424428    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cce9724-afd3-4690-966a-1fd6f2cd0251-kube-api-access-wsx62" (OuterVolumeSpecName: "kube-api-access-wsx62") pod "8cce9724-afd3-4690-966a-1fd6f2cd0251" (UID: "8cce9724-afd3-4690-966a-1fd6f2cd0251"). InnerVolumeSpecName "kube-api-access-wsx62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:52:01.424551 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.424535    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92692b23-1a8a-4890-8af7-e9c8da0e17b5-kube-api-access-xrk2p" (OuterVolumeSpecName: "kube-api-access-xrk2p") pod "92692b23-1a8a-4890-8af7-e9c8da0e17b5" (UID: "92692b23-1a8a-4890-8af7-e9c8da0e17b5"). InnerVolumeSpecName "kube-api-access-xrk2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:52:01.523274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.523250    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrk2p\" (UniqueName: \"kubernetes.io/projected/92692b23-1a8a-4890-8af7-e9c8da0e17b5-kube-api-access-xrk2p\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:52:01.523274 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.523270    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cce9724-afd3-4690-966a-1fd6f2cd0251-plugins-config-volume\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:52:01.523416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.523279    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsx62\" (UniqueName: \"kubernetes.io/projected/8cce9724-afd3-4690-966a-1fd6f2cd0251-kube-api-access-wsx62\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:52:01.626164 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.626111    2563 generic.go:356] "Generic (PLEG): container finished" podID="92692b23-1a8a-4890-8af7-e9c8da0e17b5" containerID="868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e" exitCode=0
Sep 11 18:52:01.626164 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.626152    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"
Sep 11 18:52:01.626336 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.626204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5" event={"ID":"92692b23-1a8a-4890-8af7-e9c8da0e17b5","Type":"ContainerDied","Data":"868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e"}
Sep 11 18:52:01.626336 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.626265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5" event={"ID":"92692b23-1a8a-4890-8af7-e9c8da0e17b5","Type":"ContainerDied","Data":"ec355e8c6a80989ea9b2b6d877e23eda4f0311d4386d2400f5918043500b2e5e"}
Sep 11 18:52:01.626336 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.626297    2563 scope.go:117] "RemoveContainer" containerID="868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e"
Sep 11 18:52:01.627645 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.627624    2563 generic.go:356] "Generic (PLEG): container finished" podID="8cce9724-afd3-4690-966a-1fd6f2cd0251" containerID="7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe" exitCode=0
Sep 11 18:52:01.627765 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.627678    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"
Sep 11 18:52:01.627765 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.627678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz" event={"ID":"8cce9724-afd3-4690-966a-1fd6f2cd0251","Type":"ContainerDied","Data":"7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe"}
Sep 11 18:52:01.627881 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.627793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz" event={"ID":"8cce9724-afd3-4690-966a-1fd6f2cd0251","Type":"ContainerDied","Data":"729fbb775b18146ad27912485a22ebb9c5cea42885e62049f4ac8cc79c0e46a0"}
Sep 11 18:52:01.635520 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.635504    2563 scope.go:117] "RemoveContainer" containerID="868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e"
Sep 11 18:52:01.635831 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:52:01.635811    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e\": container with ID starting with 868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e not found: ID does not exist" containerID="868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e"
Sep 11 18:52:01.635881 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.635838    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e"} err="failed to get container status \"868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e\": rpc error: code = NotFound desc = could not find container \"868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e\": container with ID starting with 868fc961d89ce47c4aa9f0ad06db5aa8bf620b6f1a96aeb60822e4d45593122e not found: ID does not exist"
Sep 11 18:52:01.635881 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.635854    2563 scope.go:117] "RemoveContainer" containerID="7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe"
Sep 11 18:52:01.645233 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.645210    2563 scope.go:117] "RemoveContainer" containerID="7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe"
Sep 11 18:52:01.645667 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:52:01.645644    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe\": container with ID starting with 7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe not found: ID does not exist" containerID="7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe"
Sep 11 18:52:01.645735 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.645671    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe"} err="failed to get container status \"7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe\": rpc error: code = NotFound desc = could not find container \"7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe\": container with ID starting with 7644e342433872676da287b64e6b04cd4f9084b8bc845404d256cb354a793abe not found: ID does not exist"
Sep 11 18:52:01.651019 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.651000    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"]
Sep 11 18:52:01.653845 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.653824    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6c4cc-6cf9f88fb9-mdftz"]
Sep 11 18:52:01.670439 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.670277    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"]
Sep 11 18:52:01.672060 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:01.672043    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6c4cc-98fb9fcd-tskb5"]
Sep 11 18:52:03.111876 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:03.111842    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cce9724-afd3-4690-966a-1fd6f2cd0251" path="/var/lib/kubelet/pods/8cce9724-afd3-4690-966a-1fd6f2cd0251/volumes"
Sep 11 18:52:03.112218 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:03.112184    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92692b23-1a8a-4890-8af7-e9c8da0e17b5" path="/var/lib/kubelet/pods/92692b23-1a8a-4890-8af7-e9c8da0e17b5/volumes"
Sep 11 18:52:03.112483 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:03.112469    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="971e1052-d1d1-4f4b-a13e-a0419bc7f360" path="/var/lib/kubelet/pods/971e1052-d1d1-4f4b-a13e-a0419bc7f360/volumes"
Sep 11 18:52:03.657145 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:03.657112    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:52:12.360334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:12.360301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:52:45.415755 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:52:45.415725    2563 scope.go:117] "RemoveContainer" containerID="84eded03f613aeffdaa0788a2af291ff79ea82ccaf3082fb1a8743b65254378f"
Sep 11 18:53:34.645008 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:53:34.644970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:54:00.237449 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:54:00.237416    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:54:08.152622 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:54:08.152587    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:54:15.854195 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:54:15.854166    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:54:33.854236 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:54:33.854203    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:54:39.748707 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:54:39.748526    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:55:45.246511 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:55:45.246486    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:55:45.254393 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:55:45.254355    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:55:45.255388 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:55:45.255348    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:55:45.262983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:55:45.262967    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-442rc_8248d578-0ff4-4992-b9b0-3327f5d8e663/ovn-acl-logging/0.log"
Sep 11 18:56:05.845449 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:05.845413    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:56:13.946793 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:13.946717    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:56:26.751638 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:26.751605    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:56:31.045900 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.045863    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"]
Sep 11 18:56:31.046334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.046219    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971e1052-d1d1-4f4b-a13e-a0419bc7f360" containerName="revoke-keys"
Sep 11 18:56:31.046334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.046232    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="971e1052-d1d1-4f4b-a13e-a0419bc7f360" containerName="revoke-keys"
Sep 11 18:56:31.046334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.046245    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92692b23-1a8a-4890-8af7-e9c8da0e17b5" containerName="payload-pre-processing"
Sep 11 18:56:31.046334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.046252    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="92692b23-1a8a-4890-8af7-e9c8da0e17b5" containerName="payload-pre-processing"
Sep 11 18:56:31.046334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.046260    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cce9724-afd3-4690-966a-1fd6f2cd0251" containerName="payload-processing"
Sep 11 18:56:31.046334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.046265    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cce9724-afd3-4690-966a-1fd6f2cd0251" containerName="payload-processing"
Sep 11 18:56:31.046334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.046322    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cce9724-afd3-4690-966a-1fd6f2cd0251" containerName="payload-processing"
Sep 11 18:56:31.046334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.046330    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="92692b23-1a8a-4890-8af7-e9c8da0e17b5" containerName="payload-pre-processing"
Sep 11 18:56:31.046334 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.046337    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="971e1052-d1d1-4f4b-a13e-a0419bc7f360" containerName="revoke-keys"
Sep 11 18:56:31.049291 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.049273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.051793 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.051773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-22c08770-kserve-self-signed-certs\""
Sep 11 18:56:31.061367 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.061332    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"]
Sep 11 18:56:31.213584 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.213561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-dshm\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.213729 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.213671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-home\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.213729 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.213698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-tmp-dir\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.213729 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.213719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7549\" (UniqueName: \"kubernetes.io/projected/ae08de08-fcae-44be-97ad-48a64727e23c-kube-api-access-q7549\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.213901 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.213758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-model-cache\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.213901 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.213781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae08de08-fcae-44be-97ad-48a64727e23c-tls-certs\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.314313 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.314249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-home\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.314313 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.314281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-tmp-dir\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.314313 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.314308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7549\" (UniqueName: \"kubernetes.io/projected/ae08de08-fcae-44be-97ad-48a64727e23c-kube-api-access-q7549\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.314598 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.314333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-model-cache\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.314598 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.314447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae08de08-fcae-44be-97ad-48a64727e23c-tls-certs\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.314598 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.314506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-dshm\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.314768 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.314697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-tmp-dir\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.314768 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.314733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-model-cache\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.314859 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.314784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-home\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.316681 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.316661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-dshm\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.316952 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.316932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae08de08-fcae-44be-97ad-48a64727e23c-tls-certs\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.322325 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.322305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7549\" (UniqueName: \"kubernetes.io/projected/ae08de08-fcae-44be-97ad-48a64727e23c-kube-api-access-q7549\") pod \"e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") " pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.365802 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.365779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:31.638480 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.638453    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"]
Sep 11 18:56:31.640489 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:56:31.640460    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae08de08_fcae_44be_97ad_48a64727e23c.slice/crio-c58258421616a90605bd7dd9b2fcf9b4d6dc03e5e2a3d5a2dc5575ac49aba242 WatchSource:0}: Error finding container c58258421616a90605bd7dd9b2fcf9b4d6dc03e5e2a3d5a2dc5575ac49aba242: Status 404 returned error can't find the container with id c58258421616a90605bd7dd9b2fcf9b4d6dc03e5e2a3d5a2dc5575ac49aba242
Sep 11 18:56:31.840804 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.840770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh" event={"ID":"ae08de08-fcae-44be-97ad-48a64727e23c","Type":"ContainerStarted","Data":"e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294"}
Sep 11 18:56:31.840980 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.840816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh" event={"ID":"ae08de08-fcae-44be-97ad-48a64727e23c","Type":"ContainerStarted","Data":"c58258421616a90605bd7dd9b2fcf9b4d6dc03e5e2a3d5a2dc5575ac49aba242"}
Sep 11 18:56:31.858958 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:31.858917    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh" podStartSLOduration=0.85890385 podStartE2EDuration="858.90385ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:56:31.857989879 +0000 UTC m=+2447.252504902" watchObservedRunningTime="2026-09-11 18:56:31.85890385 +0000 UTC m=+2447.253418888"
Sep 11 18:56:34.955960 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:34.955919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:56:41.366303 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:41.366272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:41.366790 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:41.366311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:41.378531 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:41.378501    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:56:41.896308 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:56:41.896277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:57:11.985022 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:11.984987    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"]
Sep 11 18:57:12.856261 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:12.856231    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:57:19.781120 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:19.781088    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:57:25.442851 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:25.442814    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:57:27.003454 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.003395    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh" podUID="ae08de08-fcae-44be-97ad-48a64727e23c" containerName="main" containerID="cri-o://e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294" gracePeriod=45
Sep 11 18:57:27.245872 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.245852    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:57:27.360150 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.360095    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7549\" (UniqueName: \"kubernetes.io/projected/ae08de08-fcae-44be-97ad-48a64727e23c-kube-api-access-q7549\") pod \"ae08de08-fcae-44be-97ad-48a64727e23c\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") "
Sep 11 18:57:27.360268 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.360151    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae08de08-fcae-44be-97ad-48a64727e23c-tls-certs\") pod \"ae08de08-fcae-44be-97ad-48a64727e23c\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") "
Sep 11 18:57:27.360268 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.360191    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-tmp-dir\") pod \"ae08de08-fcae-44be-97ad-48a64727e23c\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") "
Sep 11 18:57:27.360268 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.360224    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-home\") pod \"ae08de08-fcae-44be-97ad-48a64727e23c\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") "
Sep 11 18:57:27.360469 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.360266    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-model-cache\") pod \"ae08de08-fcae-44be-97ad-48a64727e23c\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") "
Sep 11 18:57:27.360469 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.360320    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-dshm\") pod \"ae08de08-fcae-44be-97ad-48a64727e23c\" (UID: \"ae08de08-fcae-44be-97ad-48a64727e23c\") "
Sep 11 18:57:27.360573 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.360510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae08de08-fcae-44be-97ad-48a64727e23c" (UID: "ae08de08-fcae-44be-97ad-48a64727e23c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:57:27.360624 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.360575    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-home" (OuterVolumeSpecName: "home") pod "ae08de08-fcae-44be-97ad-48a64727e23c" (UID: "ae08de08-fcae-44be-97ad-48a64727e23c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:57:27.360624 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.360600    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae08de08-fcae-44be-97ad-48a64727e23c" (UID: "ae08de08-fcae-44be-97ad-48a64727e23c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:57:27.360705 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.360639    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-tmp-dir\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:57:27.362306 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.362279    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-dshm" (OuterVolumeSpecName: "dshm") pod "ae08de08-fcae-44be-97ad-48a64727e23c" (UID: "ae08de08-fcae-44be-97ad-48a64727e23c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:57:27.362434 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.362357    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae08de08-fcae-44be-97ad-48a64727e23c-kube-api-access-q7549" (OuterVolumeSpecName: "kube-api-access-q7549") pod "ae08de08-fcae-44be-97ad-48a64727e23c" (UID: "ae08de08-fcae-44be-97ad-48a64727e23c"). InnerVolumeSpecName "kube-api-access-q7549". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:57:27.362434 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.362359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae08de08-fcae-44be-97ad-48a64727e23c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae08de08-fcae-44be-97ad-48a64727e23c" (UID: "ae08de08-fcae-44be-97ad-48a64727e23c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:57:27.460998 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.460971    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-home\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:57:27.461100 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.461008    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-model-cache\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:57:27.461100 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.461024    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae08de08-fcae-44be-97ad-48a64727e23c-dshm\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:57:27.461100 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.461037    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7549\" (UniqueName: \"kubernetes.io/projected/ae08de08-fcae-44be-97ad-48a64727e23c-kube-api-access-q7549\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:57:27.461100 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:27.461050    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae08de08-fcae-44be-97ad-48a64727e23c-tls-certs\") on node \"ip-10-0-143-92.ec2.internal\" DevicePath \"\""
Sep 11 18:57:28.098513 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:28.098484    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae08de08-fcae-44be-97ad-48a64727e23c" containerID="e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294" exitCode=0
Sep 11 18:57:28.098908 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:28.098540    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"
Sep 11 18:57:28.098908 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:28.098548    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh" event={"ID":"ae08de08-fcae-44be-97ad-48a64727e23c","Type":"ContainerDied","Data":"e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294"}
Sep 11 18:57:28.098908 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:28.098599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh" event={"ID":"ae08de08-fcae-44be-97ad-48a64727e23c","Type":"ContainerDied","Data":"c58258421616a90605bd7dd9b2fcf9b4d6dc03e5e2a3d5a2dc5575ac49aba242"}
Sep 11 18:57:28.098908 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:28.098616    2563 scope.go:117] "RemoveContainer" containerID="e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294"
Sep 11 18:57:28.107424 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:28.107407    2563 scope.go:117] "RemoveContainer" containerID="e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294"
Sep 11 18:57:28.107732 ip-10-0-143-92 kubenswrapper[2563]: E0911 18:57:28.107706    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294\": container with ID starting with e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294 not found: ID does not exist" containerID="e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294"
Sep 11 18:57:28.107837 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:28.107745    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294"} err="failed to get container status \"e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294\": rpc error: code = NotFound desc = could not find container \"e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294\": container with ID starting with e0a36ed5cdff18c73a94aeb86451ebe11e5ea6ef6f3ff6787c82bf179e9bf294 not found: ID does not exist"
Sep 11 18:57:28.119157 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:28.119134    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"]
Sep 11 18:57:28.124536 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:28.124516    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-22c08770-kserve-5b7b889455-l95mh"]
Sep 11 18:57:29.114530 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:29.114489    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae08de08-fcae-44be-97ad-48a64727e23c" path="/var/lib/kubelet/pods/ae08de08-fcae-44be-97ad-48a64727e23c/volumes"
Sep 11 18:57:32.263197 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:32.263162    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:57:39.636485 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:39.636409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:57:40.407954 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.407919    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"]
Sep 11 18:57:40.408405 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.408361    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae08de08-fcae-44be-97ad-48a64727e23c" containerName="main"
Sep 11 18:57:40.408405 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.408396    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae08de08-fcae-44be-97ad-48a64727e23c" containerName="main"
Sep 11 18:57:40.408567 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.408509    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae08de08-fcae-44be-97ad-48a64727e23c" containerName="main"
Sep 11 18:57:40.412667 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.412644    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.419923 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.419602    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lvtx6\""
Sep 11 18:57:40.419923 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.419707    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:57:40.419923 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.419602    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:57:40.419923 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.419810    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:57:40.419923 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.419713    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 18:57:40.425366 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.425342    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"]
Sep 11 18:57:40.567173 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.567149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qkrl\" (UniqueName: \"kubernetes.io/projected/b3a1faa4-31a6-450b-9013-0c89a7f197ea-kube-api-access-6qkrl\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.567299 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.567200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3a1faa4-31a6-450b-9013-0c89a7f197ea-maas-api-tls\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.567299 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.567234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3a1faa4-31a6-450b-9013-0c89a7f197ea-metrics-certs\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.567299 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.567281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a1faa4-31a6-450b-9013-0c89a7f197ea-openshift-service-ca\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.668329 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.668269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3a1faa4-31a6-450b-9013-0c89a7f197ea-metrics-certs\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.668329 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.668309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a1faa4-31a6-450b-9013-0c89a7f197ea-openshift-service-ca\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.668683 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.668431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qkrl\" (UniqueName: \"kubernetes.io/projected/b3a1faa4-31a6-450b-9013-0c89a7f197ea-kube-api-access-6qkrl\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.668683 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.668521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3a1faa4-31a6-450b-9013-0c89a7f197ea-maas-api-tls\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.669052 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.669032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a1faa4-31a6-450b-9013-0c89a7f197ea-openshift-service-ca\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.670779 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.670756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3a1faa4-31a6-450b-9013-0c89a7f197ea-metrics-certs\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.670859 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.670824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3a1faa4-31a6-450b-9013-0c89a7f197ea-maas-api-tls\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.676776 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.676755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qkrl\" (UniqueName: \"kubernetes.io/projected/b3a1faa4-31a6-450b-9013-0c89a7f197ea-kube-api-access-6qkrl\") pod \"maas-api-8559f566dc-t7ct7\" (UID: \"b3a1faa4-31a6-450b-9013-0c89a7f197ea\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.723921 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.723896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:40.993380 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.993333    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"]
Sep 11 18:57:40.994721 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:57:40.994697    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a1faa4_31a6_450b_9013_0c89a7f197ea.slice/crio-e1ca5327596ac99edcaea61176007433fc5246c72b84c45892aa6e15e0f04ee2 WatchSource:0}: Error finding container e1ca5327596ac99edcaea61176007433fc5246c72b84c45892aa6e15e0f04ee2: Status 404 returned error can't find the container with id e1ca5327596ac99edcaea61176007433fc5246c72b84c45892aa6e15e0f04ee2
Sep 11 18:57:40.996416 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:40.996399    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:57:41.160663 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:41.160634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7" event={"ID":"b3a1faa4-31a6-450b-9013-0c89a7f197ea","Type":"ContainerStarted","Data":"e1ca5327596ac99edcaea61176007433fc5246c72b84c45892aa6e15e0f04ee2"}
Sep 11 18:57:42.166718 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:42.166685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7" event={"ID":"b3a1faa4-31a6-450b-9013-0c89a7f197ea","Type":"ContainerStarted","Data":"18b1d3b7e0f3aecf904e90032607bf32c56df2e0c4a2f7f69cba2383b38dc6aa"}
Sep 11 18:57:42.167109 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:42.166733    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:42.184651 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:42.184608    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7" podStartSLOduration=1.778133693 podStartE2EDuration="2.184578775s" totalImagesPullingTime="406.445082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:57:42.184306438 +0000 UTC m=+2517.578821500" watchObservedRunningTime="2026-09-11 18:57:42.184578775 +0000 UTC m=+2517.579093796"
Sep 11 18:57:46.644002 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:46.643972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:57:48.176318 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:48.176292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8559f566dc-t7ct7"
Sep 11 18:57:52.446362 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:52.446331    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:57:59.550505 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:57:59.550472    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xv98x"]
Sep 11 18:58:01.882198 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:01.882161    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76687479d9-nl2b9_3c98f310-1de5-4352-8d5f-41d014340277/authorino/0.log"
Sep 11 18:58:07.730463 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:07.730430    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f4cbcd7d-lx98r_4c116a8f-4a34-4598-8347-f1d84eb51851/copy-manifests/0.log"
Sep 11 18:58:07.737542 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:07.737518    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f4cbcd7d-lx98r_4c116a8f-4a34-4598-8347-f1d84eb51851/manager/0.log"
Sep 11 18:58:08.281074 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:08.281042    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8559f566dc-t7ct7_b3a1faa4-31a6-450b-9013-0c89a7f197ea/maas-api/0.log"
Sep 11 18:58:08.392270 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:08.392250    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819190-dkgpw_56eebf66-9d7a-44ac-a4f4-e7d4c3cdf6da/cleanup/0.log"
Sep 11 18:58:09.350511 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:09.350481    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76687479d9-nl2b9_3c98f310-1de5-4352-8d5f-41d014340277/authorino/0.log"
Sep 11 18:58:09.891158 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:09.891128    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-xv98x_3495b1cb-0e6d-4183-b202-9b307262e441/limitador/0.log"
Sep 11 18:58:10.013404 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:10.013340    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-2hjnl_06aa86dd-e6e0-426b-93a7-d258487d21e1/manager/0.log"
Sep 11 18:58:10.684788 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:10.684756    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78f85fb5d-hhxvv_91e8d4a5-d3bc-4b60-83d3-43802ce56c06/kube-auth-proxy/0.log"
Sep 11 18:58:11.142751 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:11.142673    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dbccf7754-qnh7w_4c33a960-4be6-4055-9855-b13ccf170c79/router/0.log"
Sep 11 18:58:11.928209 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:11.928177    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzg6zp_2e628f0d-70d0-4397-b6c4-298b4dab91e5/main/0.log"
Sep 11 18:58:12.034892 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:12.034870    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jsshp_0f92b956-3605-4b83-b672-8c9f5c774f9f/main/0.log"
Sep 11 18:58:12.143077 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:12.143049    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-fvp57_b5ed16e3-328f-4143-9d0e-200121d00c4d/main/0.log"
Sep 11 18:58:20.745906 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:20.745877    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-phxl4_693fe3c2-fe4c-4e69-9922-8b23c60e5d5e/global-pull-secret-syncer/0.log"
Sep 11 18:58:20.843527 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:20.843501    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n8qwz_cc5cc477-94d4-4770-8901-f1e8813f8ebc/konnectivity-agent/0.log"
Sep 11 18:58:20.945405 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:20.945359    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-92.ec2.internal_3d29890478aea68f0049d36cf8ba8661/haproxy/0.log"
Sep 11 18:58:26.024063 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:26.024033    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76687479d9-nl2b9_3c98f310-1de5-4352-8d5f-41d014340277/authorino/0.log"
Sep 11 18:58:26.259145 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:26.259085    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-xv98x_3495b1cb-0e6d-4183-b202-9b307262e441/limitador/0.log"
Sep 11 18:58:26.353053 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:26.353023    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-2hjnl_06aa86dd-e6e0-426b-93a7-d258487d21e1/manager/0.log"
Sep 11 18:58:28.190220 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.190186    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-59dbc9dffc-rmjfk_7370ff00-1076-43b2-a222-ed33ef415994/metrics-server/0.log"
Sep 11 18:58:28.407361 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.407335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbjjb_ef955b9e-81c8-4611-baa6-29c300923c7f/node-exporter/0.log"
Sep 11 18:58:28.427989 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.427966    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbjjb_ef955b9e-81c8-4611-baa6-29c300923c7f/kube-rbac-proxy/0.log"
Sep 11 18:58:28.447232 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.447212    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbjjb_ef955b9e-81c8-4611-baa6-29c300923c7f/init-textfile/0.log"
Sep 11 18:58:28.583987 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.583921    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e23f5a4-4c52-4091-91c4-127c8c834e08/prometheus/0.log"
Sep 11 18:58:28.604768 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.604735    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e23f5a4-4c52-4091-91c4-127c8c834e08/config-reloader/0.log"
Sep 11 18:58:28.625240 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.625216    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e23f5a4-4c52-4091-91c4-127c8c834e08/thanos-sidecar/0.log"
Sep 11 18:58:28.643983 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.643966    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e23f5a4-4c52-4091-91c4-127c8c834e08/kube-rbac-proxy-web/0.log"
Sep 11 18:58:28.663970 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.663950    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e23f5a4-4c52-4091-91c4-127c8c834e08/kube-rbac-proxy/0.log"
Sep 11 18:58:28.683314 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.683286    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e23f5a4-4c52-4091-91c4-127c8c834e08/kube-rbac-proxy-thanos/0.log"
Sep 11 18:58:28.702915 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.702890    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e23f5a4-4c52-4091-91c4-127c8c834e08/init-config-reloader/0.log"
Sep 11 18:58:28.886922 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.886871    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/thanos-query/0.log"
Sep 11 18:58:28.903388 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.903360    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/kube-rbac-proxy-web/0.log"
Sep 11 18:58:28.922401 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.922360    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/kube-rbac-proxy/0.log"
Sep 11 18:58:28.941422 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.941404    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/prom-label-proxy/0.log"
Sep 11 18:58:28.963484 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.963464    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/kube-rbac-proxy-rules/0.log"
Sep 11 18:58:28.981725 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:28.981707    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79446bcfd8-5xjmk_87597028-7fc4-4cdd-aeb2-317648c732c0/kube-rbac-proxy-metrics/0.log"
Sep 11 18:58:30.383212 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.383178    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"]
Sep 11 18:58:30.386745 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.386727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.389417 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.389391    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9k92\"/\"openshift-service-ca.crt\""
Sep 11 18:58:30.389572 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.389419    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9k92\"/\"kube-root-ca.crt\""
Sep 11 18:58:30.389572 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.389391    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r9k92\"/\"default-dockercfg-bfrpt\""
Sep 11 18:58:30.396599 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.396573    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"]
Sep 11 18:58:30.541741 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.541717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-proc\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.541859 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.541764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-sys\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.541859 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.541783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-podres\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.541943 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.541853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkrp\" (UniqueName: \"kubernetes.io/projected/912aedf1-0ead-455b-abaa-735a1b5aeb8f-kube-api-access-sqkrp\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.541943 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.541889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-lib-modules\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.642719 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.642662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkrp\" (UniqueName: \"kubernetes.io/projected/912aedf1-0ead-455b-abaa-735a1b5aeb8f-kube-api-access-sqkrp\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.642719 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.642691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-lib-modules\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.642905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.642726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-proc\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.642905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.642767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-sys\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.642905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.642787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-podres\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.642905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.642868    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-proc\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.642905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.642885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-sys\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.642905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.642869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-lib-modules\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.642905 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.642897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/912aedf1-0ead-455b-abaa-735a1b5aeb8f-podres\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.652538 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.652510    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkrp\" (UniqueName: \"kubernetes.io/projected/912aedf1-0ead-455b-abaa-735a1b5aeb8f-kube-api-access-sqkrp\") pod \"perf-node-gather-daemonset-6mkvg\" (UID: \"912aedf1-0ead-455b-abaa-735a1b5aeb8f\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.697535 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.697513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:30.782106 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.782085    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/2.log"
Sep 11 18:58:30.791431 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.791405    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lgjg8_fdd54bfd-280d-48a9-bdaf-4a38738c6856/console-operator/3.log"
Sep 11 18:58:30.968936 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:30.968915    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"]
Sep 11 18:58:30.970710 ip-10-0-143-92 kubenswrapper[2563]: W0911 18:58:30.970682    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod912aedf1_0ead_455b_abaa_735a1b5aeb8f.slice/crio-e162736a9615683d080602a262c01fc6eb0b4ab8dc4ed829317889bbc84e2924 WatchSource:0}: Error finding container e162736a9615683d080602a262c01fc6eb0b4ab8dc4ed829317889bbc84e2924: Status 404 returned error can't find the container with id e162736a9615683d080602a262c01fc6eb0b4ab8dc4ed829317889bbc84e2924
Sep 11 18:58:31.389111 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:31.389037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg" event={"ID":"912aedf1-0ead-455b-abaa-735a1b5aeb8f","Type":"ContainerStarted","Data":"209a8580e3c41584959871c990412b8f2da006f445a9f1fc979bd251e6f1c6c7"}
Sep 11 18:58:31.389111 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:31.389099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg" event={"ID":"912aedf1-0ead-455b-abaa-735a1b5aeb8f","Type":"ContainerStarted","Data":"e162736a9615683d080602a262c01fc6eb0b4ab8dc4ed829317889bbc84e2924"}
Sep 11 18:58:31.389473 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:31.389131    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:31.418778 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:31.418736    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg" podStartSLOduration=1.418721149 podStartE2EDuration="1.418721149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:58:31.415834281 +0000 UTC m=+2566.810349314" watchObservedRunningTime="2026-09-11 18:58:31.418721149 +0000 UTC m=+2566.813236174"
Sep 11 18:58:33.331816 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:33.331785    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d672m_f4fa4956-4afa-42b9-b95f-94cb9cc018df/dns/0.log"
Sep 11 18:58:33.357793 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:33.357762    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d672m_f4fa4956-4afa-42b9-b95f-94cb9cc018df/kube-rbac-proxy/0.log"
Sep 11 18:58:33.481537 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:33.481515    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qfkcq_3725edf8-bf10-4f95-99cb-2dc2581932a4/dns-node-resolver/0.log"
Sep 11 18:58:34.126029 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:34.126003    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zkrtp_9b80a3a2-8a1b-4382-a49f-ac7a667d5aa0/node-ca/0.log"
Sep 11 18:58:35.519313 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:35.519282    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78f85fb5d-hhxvv_91e8d4a5-d3bc-4b60-83d3-43802ce56c06/kube-auth-proxy/0.log"
Sep 11 18:58:35.651228 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:35.651202    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dbccf7754-qnh7w_4c33a960-4be6-4055-9855-b13ccf170c79/router/0.log"
Sep 11 18:58:36.551939 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:36.551913    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tw4p9_4e8d1a68-a884-44ec-b9cc-f6ad7f74156c/serve-healthcheck-canary/0.log"
Sep 11 18:58:37.037481 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:37.037457    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-bmtcc_f7d9182c-e4a7-4850-b8f9-a43199335b21/insights-operator/0.log"
Sep 11 18:58:37.041287 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:37.041259    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-bmtcc_f7d9182c-e4a7-4850-b8f9-a43199335b21/insights-operator/1.log"
Sep 11 18:58:37.202874 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:37.202846    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zw5qz_3c112901-362e-4876-9fdd-155b74b211a5/kube-rbac-proxy/0.log"
Sep 11 18:58:37.220637 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:37.220618    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zw5qz_3c112901-362e-4876-9fdd-155b74b211a5/exporter/0.log"
Sep 11 18:58:37.239997 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:37.239978    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zw5qz_3c112901-362e-4876-9fdd-155b74b211a5/extractor/0.log"
Sep 11 18:58:37.402437 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:37.402366    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-6mkvg"
Sep 11 18:58:40.197341 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:40.197308    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f4cbcd7d-lx98r_4c116a8f-4a34-4598-8347-f1d84eb51851/manager/0.log"
Sep 11 18:58:40.216694 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:40.216673    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f4cbcd7d-lx98r_4c116a8f-4a34-4598-8347-f1d84eb51851/copy-manifests/0.log"
Sep 11 18:58:41.549891 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:41.549861    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68f8b56d64-9gd99_77101585-1220-42ec-b540-98268146dc48/manager/0.log"
Sep 11 18:58:42.063041 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:42.063016    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78f85fb5d-hhxvv_91e8d4a5-d3bc-4b60-83d3-43802ce56c06/kube-auth-proxy/0.log"
Sep 11 18:58:42.165347 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:42.165329    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dbccf7754-qnh7w_4c33a960-4be6-4055-9855-b13ccf170c79/router/0.log"
Sep 11 18:58:44.075783 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:44.075753    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78f85fb5d-hhxvv_91e8d4a5-d3bc-4b60-83d3-43802ce56c06/kube-auth-proxy/0.log"
Sep 11 18:58:44.198609 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:44.198578    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dbccf7754-qnh7w_4c33a960-4be6-4055-9855-b13ccf170c79/router/0.log"
Sep 11 18:58:45.475529 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:45.475499    2563 scope.go:117] "RemoveContainer" containerID="23ed19e9894e12a405f8a68f37ef38eae5949c857dba30d390843879c437c0fb"
Sep 11 18:58:46.933024 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:46.932999    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-sjk9h_ab1abbea-be88-42ea-a22b-4ac4669e8f40/migrator/0.log"
Sep 11 18:58:46.950669 ip-10-0-143-92 kubenswrapper[2563]: I0911 18:58:46.950641    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-sjk9h_ab1abbea-be88-42ea-a22b-4ac4669e8f40/graceful-termination/0.log"