Sep 09 12:46:27.507103 ip-10-0-140-133 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 12:46:27.507108 ip-10-0-140-133 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 12:46:27.507115 ip-10-0-140-133 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 12:46:27.507269 ip-10-0-140-133 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 12:46:38.770275 ip-10-0-140-133 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 12:46:38.770289 ip-10-0-140-133 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6cbee944958047638f2ed8e7e081d1c3 --
Sep 09 12:48:36.958098 ip-10-0-140-133 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 12:48:37.374530 ip-10-0-140-133 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 12:48:37.374530 ip-10-0-140-133 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 12:48:37.374530 ip-10-0-140-133 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 12:48:37.374530 ip-10-0-140-133 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 12:48:37.374530 ip-10-0-140-133 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 12:48:37.375249 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.375119    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 12:48:37.380881 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380862    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:48:37.380881 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380881    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380884    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380888    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380891    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380894    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380896    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380899    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380902    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380905    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380908    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380911    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380914    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380916    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380919    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380922    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380924    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380927    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380930    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380932    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380935    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380938    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:48:37.380952 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380941    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380944    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380947    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380949    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380952    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380954    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380957    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380960    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380963    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380966    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380969    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380971    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380974    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380976    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380979    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380981    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380984    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380986    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380988    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:48:37.381480 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380991    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.380993    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381004    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381008    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381011    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381014    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381016    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381020    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381024    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381027    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381029    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381032    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381034    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381037    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381040    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381043    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381046    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381048    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381051    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:48:37.381957 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381054    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381057    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381066    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381069    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381071    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381074    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381077    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381079    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381082    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381085    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381088    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381093    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381096    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381098    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381101    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381103    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381106    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381108    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381111    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:48:37.382425 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381113    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381116    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381118    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381121    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381124    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381126    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381130    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381134    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381136    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381140    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381143    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381146    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381149    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381152    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381155    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381158    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381160    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381163    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381165    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381168    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:48:37.382920 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381890    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381898    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381901    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381904    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381906    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381910    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381912    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381915    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381918    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381920    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381923    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381925    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381928    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381930    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381933    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381937    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381940    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381943    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381945    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:48:37.383421 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381948    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381950    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381954    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381957    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381959    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381962    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381965    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381969    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381972    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381975    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381978    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381980    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381982    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381985    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381987    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381990    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381992    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381995    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.381997    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382001    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:48:37.383901 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382003    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382006    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382008    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382011    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382013    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382016    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382018    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382021    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382025    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382029    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382031    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382034    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382036    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382039    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382042    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382045    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382047    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382050    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382052    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382054    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:48:37.384388 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382058    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382061    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382063    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382066    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382069    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382071    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382074    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382077    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382079    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382081    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382084    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382086    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382090    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382092    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382095    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382098    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382101    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382103    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382106    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382108    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:48:37.384894 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382111    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382113    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382116    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382118    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382121    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382124    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382127    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382129    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382132    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382135    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382137    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382140    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382142    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382145    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382147    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382150    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382152    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382155    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382157    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:48:37.385491 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382160    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382563    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382568    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382573    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382577    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382581    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382584    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382587    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382589    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382592    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382595    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382598    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382600    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382603    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382606    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382608    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382611    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382613    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382617    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382619    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:48:37.385970 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382622    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382625    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382627    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382630    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382632    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382635    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382638    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382640    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382643    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382645    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382647    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382650    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382652    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382654    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382657    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382659    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382662    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382664    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382667    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382670    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:48:37.386452 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382673    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382676    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382688    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382692    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382694    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382697    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382700    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382702    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382705    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382707    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382710    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382713    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382715    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382718    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382721    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382723    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382726    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382728    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382730    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382733    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:48:37.386993 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382736    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382738    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382741    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382743    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382746    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382748    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382751    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382753    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382755    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382758    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382760    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382763    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382767    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382772    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382775    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382778    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382781    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382783    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382786    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382788    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:48:37.387484 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382792    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382794    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382797    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382800    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382802    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382805    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382807    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382809    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382812    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382829    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382832    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382835    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382838    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382840    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382844    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382847    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382851    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382853    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382856    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.382859    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:48:37.388012 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.382937    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.382977    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.382985    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.382990    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.382994    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.382998    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383003    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383007    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383010    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383013    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383016    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383019    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383022    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383025    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383028    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383031    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383034    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383037    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383041    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383044    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383047    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383050    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383053    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383057    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 12:48:37.388523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383060    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383063    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383066    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383069    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383072    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383075    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383077    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383080    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383084    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383087    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383090    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383093    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383096    2580 flags.go:64] FLAG: --enable-server="true"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383099    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383105    2580 flags.go:64] FLAG: --event-burst="100"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383108    2580 flags.go:64] FLAG: --event-qps="50"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383111    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383114    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383117    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383120    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383124    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383126    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383130    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383132    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383135    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 12:48:37.389107 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383138    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383141    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383144    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383146    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383149    2580 flags.go:64] FLAG: --feature-gates=""
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383153    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383156    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383160    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383163    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383166    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383169    2580 flags.go:64] FLAG: --help="false"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383171    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383174    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383177    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383182    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383185    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383189    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383191    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383194    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383196    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383200    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383204    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383207    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383210    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 09 12:48:37.389707 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383212    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383215    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383218    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383221    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383224    2580 flags.go:64] FLAG: --lock-file=""
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383226    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383229    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383232    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383238    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383241    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383244    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383246    2580 flags.go:64] FLAG: --logging-format="text"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383249    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383252    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383255    2580 flags.go:64] FLAG: --manifest-url=""
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383258    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383263    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383266    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383272    2580 flags.go:64] FLAG: --max-pods="110"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383275    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383278    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383281    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383285    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383288    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383291    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 12:48:37.390348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383294    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383302    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383305    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383308    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383312    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383315    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383320    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383323    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383326    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383329    2580 flags.go:64] FLAG: --port="10250"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383332    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383335    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f6dd7a46d445738f"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383338    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383341    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383344    2580 flags.go:64] FLAG: --register-node="true"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383347    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383350    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383353    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383355    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383358    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383362    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383364    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383367    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383370    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383373    2580 flags.go:64] FLAG: --runonce="false"
Sep 09 12:48:37.391001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383375    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383378    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383381    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383384    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383388    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383391    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383393    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383397    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383399    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383402    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383405    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383409    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383413    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383416    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383419    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383425    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383427    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383430    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383435    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383438    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383440    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383443    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383446    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383449    2580 flags.go:64] FLAG: --v="2"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383453    2580 flags.go:64] FLAG: --version="false"
Sep 09 12:48:37.391635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383457    2580 flags.go:64] FLAG: --vmodule=""
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383461    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.383464    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383580    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383583    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383586    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383589    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383592    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383594    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383597    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383599    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383604    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383606    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383609    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383611    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383614    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383618    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383621    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383629    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:48:37.392339 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383632    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383634    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383637    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383640    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383643    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383646    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383648    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383651    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383654    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383656    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383659    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383661    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383664    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383666    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383669    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383671    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383674    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383677    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383679    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383682    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:48:37.392806 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383684    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383687    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383689    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383692    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383697    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383700    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383703    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383707    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383710    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383713    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383715    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383719    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383721    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383724    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383726    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383730    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383732    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383735    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383737    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383740    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:48:37.393308 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383742    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383745    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383747    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383750    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383752    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383755    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383757    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383760    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383762    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383765    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383768    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383771    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383773    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383776    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383778    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383780    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383784    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383786    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383789    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383791    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:48:37.393795 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383794    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383796    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383799    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383802    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383805    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383807    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383810    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383812    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383839    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383842    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383845    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383848    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383850    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383853    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383856    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383858    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383861    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383863    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383866    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383869    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:48:37.394315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383871    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383874    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.383876    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.384385    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.391965    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.391979    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.391996    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.392001    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392066    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392070    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392073    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392077    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392079    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392083    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392087    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:48:37.394804 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392091    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392094    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392097    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392100    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392102    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392106    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392108    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392111    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392114    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392116    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392119    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392125    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392128    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392131    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392133    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392135    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392138    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392140    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392143    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:48:37.395276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392146    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392148    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392150    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392153    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392156    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392158    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392161    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392164    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392167    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392169    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392171    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392174    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392176    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392180    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392182    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392184    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392187    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392189    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392192    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:48:37.395752 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392200    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392202    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392205    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392207    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392210    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392212    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392214    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392217    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392219    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392222    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392224    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392228    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392232    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392237    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392240    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392243    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392245    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392248    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392251    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:48:37.396307 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392254    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392257    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392260    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392262    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392265    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392267    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392270    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392272    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392275    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392278    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392280    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392283    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392285    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392287    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392296    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392299    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392301    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392303    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392306    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392308    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:48:37.396802 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392311    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392313    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392316    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392318    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392321    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392323    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392325    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392328    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392330    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392333    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392336    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392338    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392341    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392344    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392346    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:48:37.397525 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.392351    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392518    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392523    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392527    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392529    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392532    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392536    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392540    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392543    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392546    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392549    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392553    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392563    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392565    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392568    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392570    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392573    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392575    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392577    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:48:37.398171 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392580    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392582    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392585    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392587    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392590    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392592    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392594    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392597    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392599    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392603    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392606    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392608    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392611    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392613    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392616    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392618    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392620    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392623    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392625    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392627    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:48:37.398748 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392630    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392632    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392635    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392637    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392640    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392642    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392650    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392652    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392655    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392657    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392660    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392662    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392665    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392667    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392669    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392672    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392674    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392677    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392679    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392681    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:48:37.399315 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392684    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392692    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392695    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392698    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392700    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392703    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392705    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392707    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392710    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392712    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392715    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392717    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392719    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392722    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392724    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392726    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392729    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392731    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392735    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392743    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:48:37.399895 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392746    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392748    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392751    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392754    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392757    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392760    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392762    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392765    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392767    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392769    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392772    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392774    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392776    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392779    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392782    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392784    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392786    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392789    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392791    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392793    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:48:37.400569 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:37.392796    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:48:37.401218 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.392800    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 12:48:37.401218 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.392934    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 12:48:37.401218 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.395068    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 12:48:37.401218 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.396088    2580 server.go:1019] "Starting client certificate rotation"
Sep 09 12:48:37.401218 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.396205    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 12:48:37.401218 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.396262    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 12:48:37.401218 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.398577    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 12:48:37.402912 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.402847    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 12:48:37.411385 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.411365    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 09 12:48:37.414267 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.414248    2580 log.go:25] "Validated CRI v1 image API"
Sep 09 12:48:37.416039 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.416024    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 12:48:37.420244 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.420220    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a3dc335e-b8c9-4039-bfcc-f3e5e3ee2eb7:/dev/nvme0n1p3 b82775e7-5308-41b5-98cd-5c2841e347c7:/dev/nvme0n1p4]
Sep 09 12:48:37.420312 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.420244    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 12:48:37.423371 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.423354    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 12:48:37.425490 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.425378    2580 manager.go:216] Machine: {Timestamp:2026-09-09 12:48:37.424101027 +0000 UTC m=+0.343992361 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110500 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d2ba2dcbf1984f74148c067bd9b8c SystemUUID:ec2d2ba2-dcbf-1984-f741-48c067bd9b8c BootID:6cbee944-9580-4763-8f2e-d8e7e081d1c3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:13:e7:44:9e:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:13:e7:44:9e:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:68:a9:fb:7d:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 12:48:37.425490 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.425486    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 12:48:37.425631 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.425616    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 12:48:37.428859 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.428832    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 12:48:37.429008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.428862    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-133.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 12:48:37.429640 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.429631    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 12:48:37.429670 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.429644    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 12:48:37.429670 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.429652    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 12:48:37.430214 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.430203    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 12:48:37.431422 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.431412    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 12:48:37.436057 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.436046    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 12:48:37.436184 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.436175    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 12:48:37.436218 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.436209    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 12:48:37.436246 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.436227    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 12:48:37.436295 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.436251    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 12:48:37.436295 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.436265    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 09 12:48:37.436295 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.436276    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 12:48:37.436740 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.436727    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 12:48:37.436776 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.436747    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 12:48:37.438997 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.438973    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 12:48:37.441145 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441131    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 12:48:37.441206 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441162    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 12:48:37.441322 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441312    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 12:48:37.441366 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441327    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 12:48:37.441366 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441333    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 12:48:37.441366 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441338    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 12:48:37.441366 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441346    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 12:48:37.441366 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441351    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 12:48:37.441861 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441851    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 12:48:37.441861 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441861    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 12:48:37.441936 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441875    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 12:48:37.441936 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441880    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 12:48:37.441936 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441895    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 12:48:37.441936 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.441903    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 12:48:37.443007 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.442992    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wzkwv"
Sep 09 12:48:37.443348 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.443337    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 12:48:37.443381 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.443352    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 12:48:37.447548 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.447532    2580 server.go:1268] "Started kubelet"
Sep 09 12:48:37.447737 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.447703    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 12:48:37.447840 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.447757    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 12:48:37.448306 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.448256    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 12:48:37.448306 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.448281    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-133.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 12:48:37.448506 ip-10-0-140-133 systemd[1]: Started Kubernetes Kubelet.
Sep 09 12:48:37.448711 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.448494    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 12:48:37.448764 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.448741    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 12:48:37.451360 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.451338    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 12:48:37.451581 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.451565    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wzkwv"
Sep 09 12:48:37.453008 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.452985    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 12:48:37.454032 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.454003    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 12:48:37.455614 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.455600    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 12:48:37.455614 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.455607    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 12:48:37.455750 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.455730    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-133.ec2.internal\" not found"
Sep 09 12:48:37.456109 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.454921    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-133.ec2.internal.18d3a7edf4827af1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-133.ec2.internal,UID:ip-10-0-140-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-133.ec2.internal,},FirstTimestamp:2026-09-09 12:48:37.447498481 +0000 UTC m=+0.367389811,LastTimestamp:2026-09-09 12:48:37.447498481 +0000 UTC m=+0.367389811,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-133.ec2.internal,}"
Sep 09 12:48:37.456227 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.456137    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 12:48:37.456302 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.456189    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 12:48:37.456384 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.456374    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 12:48:37.456490 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.456471    2580 factory.go:55] Registering systemd factory
Sep 09 12:48:37.456547 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.456521    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 09 12:48:37.456580 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.456520    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 12:48:37.456630 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.456580    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 12:48:37.458295 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.458279    2580 factory.go:153] Registering CRI-O factory
Sep 09 12:48:37.458295 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.458297    2580 factory.go:223] Registration of the crio container factory successfully
Sep 09 12:48:37.458410 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.458351    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 12:48:37.458410 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.458368    2580 factory.go:103] Registering Raw factory
Sep 09 12:48:37.458410 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.458378    2580 manager.go:1196] Started watching for new ooms in manager
Sep 09 12:48:37.458746 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.458729    2580 manager.go:349] Starting recovery of all containers
Sep 09 12:48:37.465008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.464980    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:48:37.468031 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.468006    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-133.ec2.internal\" not found" node="ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.469283 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.469267    2580 manager.go:354] Recovery completed
Sep 09 12:48:37.473960 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.473947    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:48:37.476449 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.476434    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:48:37.476499 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.476465    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:48:37.476499 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.476475    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:48:37.477053 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.477039    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 12:48:37.477053 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.477051    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:48:37.477171 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.477054    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 12:48:37.477171 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.477074    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 12:48:37.478438 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.478420    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:48:37.478516 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.478453    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:48:37.478516 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.478468    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:48:37.479184 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.479172    2580 policy_none.go:49] "None policy: Start"
Sep 09 12:48:37.479220 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.479190    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 12:48:37.479220 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.479199    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 12:48:37.481280 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.481269    2580 policy_none.go:47] "Start"
Sep 09 12:48:37.536457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.522451    2580 manager.go:337] "Starting Device Plugin manager"
Sep 09 12:48:37.536457 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.522649    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 12:48:37.536457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.522659    2580 server.go:87] "Starting device plugin registration server"
Sep 09 12:48:37.536457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.522910    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 12:48:37.536457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.522920    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 12:48:37.536457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.523035    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 12:48:37.536457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.523121    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 12:48:37.536457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.523130    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 12:48:37.536457 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.523733    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 12:48:37.536457 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.523777    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-133.ec2.internal\" not found"
Sep 09 12:48:37.583711 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.583674    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 12:48:37.584985 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.584967    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 12:48:37.585052 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.585041    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 12:48:37.585101 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.585072    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 12:48:37.585148 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.585123    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 12:48:37.587490 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.587463    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:48:37.623209 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.623175    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:48:37.625038 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.624978    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:48:37.625038 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.625011    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:48:37.625038 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.625026    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:48:37.625236 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.625057    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.630976 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.630960    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.631027 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.630986    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-133.ec2.internal\": node \"ip-10-0-140-133.ec2.internal\" not found"
Sep 09 12:48:37.647506 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.647476    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-133.ec2.internal\" not found"
Sep 09 12:48:37.685368 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.685329    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-133.ec2.internal"]
Sep 09 12:48:37.685540 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.685421    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:48:37.688199 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.688173    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:48:37.688313 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.688214    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:48:37.688313 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.688229    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:48:37.690241 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.690228    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:48:37.690360 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.690345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.690413 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.690380    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:48:37.693479 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.693462    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:48:37.693572 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.693485    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:48:37.693572 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.693494    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:48:37.693572 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.693464    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:48:37.693572 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.693558    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:48:37.693692 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.693574    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:48:37.699424 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.699405    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.699504 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.699435    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:48:37.700322 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.700308    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:48:37.700389 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.700334    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:48:37.700389 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.700344    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:48:37.715536 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.715510    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-133.ec2.internal\" not found" node="ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.721410 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.721395    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-133.ec2.internal\" not found" node="ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.748228 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.748198    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-133.ec2.internal\" not found"
Sep 09 12:48:37.758531 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.758502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcbae3b2574b8b7399924363fcb3393c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal\" (UID: \"bcbae3b2574b8b7399924363fcb3393c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.758591 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.758535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5785166f811993e15a031323bc2330c5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-133.ec2.internal\" (UID: \"5785166f811993e15a031323bc2330c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.758591 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.758553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcbae3b2574b8b7399924363fcb3393c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal\" (UID: \"bcbae3b2574b8b7399924363fcb3393c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.848767 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.848723    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-133.ec2.internal\" not found"
Sep 09 12:48:37.859102 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.859078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcbae3b2574b8b7399924363fcb3393c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal\" (UID: \"bcbae3b2574b8b7399924363fcb3393c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.859159 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.859107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcbae3b2574b8b7399924363fcb3393c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal\" (UID: \"bcbae3b2574b8b7399924363fcb3393c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.859159 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.859126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5785166f811993e15a031323bc2330c5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-133.ec2.internal\" (UID: \"5785166f811993e15a031323bc2330c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.859221 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.859167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5785166f811993e15a031323bc2330c5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-133.ec2.internal\" (UID: \"5785166f811993e15a031323bc2330c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.859221 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.859206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcbae3b2574b8b7399924363fcb3393c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal\" (UID: \"bcbae3b2574b8b7399924363fcb3393c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.859281 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:37.859206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcbae3b2574b8b7399924363fcb3393c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal\" (UID: \"bcbae3b2574b8b7399924363fcb3393c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:37.949623 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:37.949544    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-133.ec2.internal\" not found"
Sep 09 12:48:38.017962 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.017922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:38.024530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.024292    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:38.050254 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.050215    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-133.ec2.internal\" not found"
Sep 09 12:48:38.150769 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.150730    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-133.ec2.internal\" not found"
Sep 09 12:48:38.251320 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.251234    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-133.ec2.internal\" not found"
Sep 09 12:48:38.255370 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.255353    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:48:38.255775 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.255759    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:38.262220 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.262202    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:48:38.266537 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.266520    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 12:48:38.267561 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.267547    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-133.ec2.internal"
Sep 09 12:48:38.275952 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.275937    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 12:48:38.395832 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.395802    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 12:48:38.396326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.395959    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 12:48:38.396326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.395955    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 12:48:38.396326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.395955    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 12:48:38.396326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.395962    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 12:48:38.437106 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.437075    2580 apiserver.go:52] "Watching apiserver"
Sep 09 12:48:38.451180 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.451156    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 12:48:38.451545 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.451523    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-2n2wj","kube-system/konnectivity-agent-qq7xl","openshift-dns/node-resolver-qp9gk","openshift-multus/multus-additional-cni-plugins-bpb85","openshift-multus/multus-k82nw","openshift-multus/network-metrics-daemon-gjfxj","openshift-network-operator/iptables-alerter-pckth","openshift-ovn-kubernetes/ovnkube-node-9cq52","kube-system/global-pull-secret-syncer-7jcs4","kube-system/kube-apiserver-proxy-ip-10-0-140-133.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr","openshift-cluster-node-tuning-operator/tuned-vc5kq","openshift-image-registry/node-ca-tgvsf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal"]
Sep 09 12:48:38.453917 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.453897    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pckth"
Sep 09 12:48:38.455914 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.455894    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 12:48:38.456031 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.456016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tgvsf"
Sep 09 12:48:38.457155 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.456716    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 12:43:37 +0000 UTC" deadline="2028-02-25 13:06:53.450827049 +0000 UTC"
Sep 09 12:48:38.457155 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.456767    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12816h18m14.994064338s"
Sep 09 12:48:38.457155 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.456913    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 12:48:38.457155 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.457094    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:48:38.457155 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.457103    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4nvpl\""
Sep 09 12:48:38.457488 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.457452    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 12:48:38.459217 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.458114    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 12:48:38.459217 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.458186    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 12:48:38.459217 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.458422    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4pllp\""
Sep 09 12:48:38.459217 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.458807    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 12:48:38.459484 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.459463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qp9gk"
Sep 09 12:48:38.461771 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.461751    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5wpd4\""
Sep 09 12:48:38.461771 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.461766    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 12:48:38.461956 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.461773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f87b0945-3613-44a4-b60b-44deb2a1143c-iptables-alerter-script\") pod \"iptables-alerter-pckth\" (UID: \"f87b0945-3613-44a4-b60b-44deb2a1143c\") " pod="openshift-network-operator/iptables-alerter-pckth"
Sep 09 12:48:38.461956 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.461804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f87b0945-3613-44a4-b60b-44deb2a1143c-host-slash\") pod \"iptables-alerter-pckth\" (UID: \"f87b0945-3613-44a4-b60b-44deb2a1143c\") " pod="openshift-network-operator/iptables-alerter-pckth"
Sep 09 12:48:38.461956 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.461847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sj97\" (UniqueName: \"kubernetes.io/projected/f87b0945-3613-44a4-b60b-44deb2a1143c-kube-api-access-8sj97\") pod \"iptables-alerter-pckth\" (UID: \"f87b0945-3613-44a4-b60b-44deb2a1143c\") " pod="openshift-network-operator/iptables-alerter-pckth"
Sep 09 12:48:38.461956 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.461870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34c5ccd1-df9d-4b19-b2ae-93d5827baef1-host\") pod \"node-ca-tgvsf\" (UID: \"34c5ccd1-df9d-4b19-b2ae-93d5827baef1\") " pod="openshift-image-registry/node-ca-tgvsf"
Sep 09 12:48:38.461956 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.461918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/34c5ccd1-df9d-4b19-b2ae-93d5827baef1-serviceca\") pod \"node-ca-tgvsf\" (UID: \"34c5ccd1-df9d-4b19-b2ae-93d5827baef1\") " pod="openshift-image-registry/node-ca-tgvsf"
Sep 09 12:48:38.461956 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.461943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ffd649fb-0687-45dd-9acc-4a8d232f144d-hosts-file\") pod \"node-resolver-qp9gk\" (UID: \"ffd649fb-0687-45dd-9acc-4a8d232f144d\") " pod="openshift-dns/node-resolver-qp9gk"
Sep 09 12:48:38.461956 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.461946    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 12:48:38.462241 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.461983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xln\" (UniqueName: \"kubernetes.io/projected/34c5ccd1-df9d-4b19-b2ae-93d5827baef1-kube-api-access-s8xln\") pod \"node-ca-tgvsf\" (UID: \"34c5ccd1-df9d-4b19-b2ae-93d5827baef1\") " pod="openshift-image-registry/node-ca-tgvsf"
Sep 09 12:48:38.462241 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.462028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd649fb-0687-45dd-9acc-4a8d232f144d-tmp-dir\") pod \"node-resolver-qp9gk\" (UID: \"ffd649fb-0687-45dd-9acc-4a8d232f144d\") " pod="openshift-dns/node-resolver-qp9gk"
Sep 09 12:48:38.462241 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.462053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxw7r\" (UniqueName: \"kubernetes.io/projected/ffd649fb-0687-45dd-9acc-4a8d232f144d-kube-api-access-wxw7r\") pod \"node-resolver-qp9gk\" (UID: \"ffd649fb-0687-45dd-9acc-4a8d232f144d\") " pod="openshift-dns/node-resolver-qp9gk"
Sep 09 12:48:38.462241 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.462094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.465590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.464541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.465590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.464655    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zjbzx\""
Sep 09 12:48:38.465590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.464752    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 12:48:38.465590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.465110    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 12:48:38.465590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.465169    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 12:48:38.465590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.465158    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 12:48:38.465590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.465362    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 12:48:38.466603 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.466584    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:38.466711 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.466651    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 12:48:38.466711 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.466672    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:48:38.467088 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.467070    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qjprr\""
Sep 09 12:48:38.468430 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.468412    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 12:48:38.469401 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.469327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:38.469467 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.469413    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:48:38.471279 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.471259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:38.471373 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.471344    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:48:38.473513 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.473493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:48:38.475809 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.475789    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 12:48:38.475927 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.475862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ckzvl\""
Sep 09 12:48:38.475927 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.475882    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 12:48:38.475927 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.475886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.478105 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.478086    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 12:48:38.478346 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.478327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.478605 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.478330    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 12:48:38.478711 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.478689    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 12:48:38.478885 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.478868    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 12:48:38.478965 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.478937    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fm9qw\""
Sep 09 12:48:38.479024 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.479012    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 12:48:38.479077 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.479041    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 12:48:38.480645 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.480622    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 12:48:38.480741 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.480708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.480845 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.480811    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 12:48:38.481382 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.481128    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t65pf\""
Sep 09 12:48:38.481382 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.481237    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 12:48:38.483040 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.483022    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 12:48:38.483124 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.483103    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:48:38.483182 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.483150    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5v55n\""
Sep 09 12:48:38.491063 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.491047    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-497pq"
Sep 09 12:48:38.501752 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.501689    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-497pq"
Sep 09 12:48:38.557065 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.557038    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 12:48:38.562421 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-sysconfig\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.562542 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-socket-dir-parent\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.562542 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-cni-netd\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.562542 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-cni-dir\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.562542 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2014baf8-52ee-420c-89d0-205989b32b87-kubelet-config\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:38.562710 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.562710 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/34c5ccd1-df9d-4b19-b2ae-93d5827baef1-serviceca\") pod \"node-ca-tgvsf\" (UID: \"34c5ccd1-df9d-4b19-b2ae-93d5827baef1\") " pod="openshift-image-registry/node-ca-tgvsf"
Sep 09 12:48:38.562710 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd649fb-0687-45dd-9acc-4a8d232f144d-tmp-dir\") pod \"node-resolver-qp9gk\" (UID: \"ffd649fb-0687-45dd-9acc-4a8d232f144d\") " pod="openshift-dns/node-resolver-qp9gk"
Sep 09 12:48:38.562710 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-run-ovn\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.562710 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562668    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-run-systemd\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.562710 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sj97\" (UniqueName: \"kubernetes.io/projected/f87b0945-3613-44a4-b60b-44deb2a1143c-kube-api-access-8sj97\") pod \"iptables-alerter-pckth\" (UID: \"f87b0945-3613-44a4-b60b-44deb2a1143c\") " pod="openshift-network-operator/iptables-alerter-pckth"
Sep 09 12:48:38.563008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ffd649fb-0687-45dd-9acc-4a8d232f144d-hosts-file\") pod \"node-resolver-qp9gk\" (UID: \"ffd649fb-0687-45dd-9acc-4a8d232f144d\") " pod="openshift-dns/node-resolver-qp9gk"
Sep 09 12:48:38.563008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8dbba43e-056a-4448-85cd-36f295e6f710-ovnkube-config\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.563008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-run-k8s-cni-cncf-io\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.563008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:38.563008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ffd649fb-0687-45dd-9acc-4a8d232f144d-hosts-file\") pod \"node-resolver-qp9gk\" (UID: \"ffd649fb-0687-45dd-9acc-4a8d232f144d\") " pod="openshift-dns/node-resolver-qp9gk"
Sep 09 12:48:38.563008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6vhk\" (UniqueName: \"kubernetes.io/projected/59e0eeed-687d-47aa-8660-d3642cb43488-kube-api-access-w6vhk\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.563008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8dbba43e-056a-4448-85cd-36f295e6f710-ovnkube-script-lib\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.563008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-sysctl-conf\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.563008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.562981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-lib-modules\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.563008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-hostroot\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd649fb-0687-45dd-9acc-4a8d232f144d-tmp-dir\") pod \"node-resolver-qp9gk\" (UID: \"ffd649fb-0687-45dd-9acc-4a8d232f144d\") " pod="openshift-dns/node-resolver-qp9gk"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-daemon-config\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2014baf8-52ee-420c-89d0-205989b32b87-dbus\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-cnibin\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/34c5ccd1-df9d-4b19-b2ae-93d5827baef1-serviceca\") pod \"node-ca-tgvsf\" (UID: \"34c5ccd1-df9d-4b19-b2ae-93d5827baef1\") " pod="openshift-image-registry/node-ca-tgvsf"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-run-ovn-kubernetes\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac2e5020-766d-4573-afa9-8c3ec3ab350c-tmp\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-run-multus-certs\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njclh\" (UniqueName: \"kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh\") pod \"network-check-target-2n2wj\" (UID: \"ab6539a3-0367-425b-9dc0-cc894772df3a\") " pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r44gn\" (UniqueName: \"kubernetes.io/projected/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-kube-api-access-r44gn\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28b841c2-6400-4ee4-bdf6-8f238cc60545-konnectivity-ca\") pod \"konnectivity-agent-qq7xl\" (UID: \"28b841c2-6400-4ee4-bdf6-8f238cc60545\") " pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8dbba43e-056a-4448-85cd-36f295e6f710-ovn-node-metrics-cert\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.563470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwlpb\" (UniqueName: \"kubernetes.io/projected/ac2e5020-766d-4573-afa9-8c3ec3ab350c-kube-api-access-fwlpb\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-etc-kubernetes\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-socket-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-run-netns\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-slash\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28b841c2-6400-4ee4-bdf6-8f238cc60545-agent-certs\") pod \"konnectivity-agent-qq7xl\" (UID: \"28b841c2-6400-4ee4-bdf6-8f238cc60545\") " pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f87b0945-3613-44a4-b60b-44deb2a1143c-iptables-alerter-script\") pod \"iptables-alerter-pckth\" (UID: \"f87b0945-3613-44a4-b60b-44deb2a1143c\") " pod="openshift-network-operator/iptables-alerter-pckth"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8dbba43e-056a-4448-85cd-36f295e6f710-env-overrides\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-system-cni-dir\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/611a1f11-6e3b-406c-97e3-5fbec3957485-cni-binary-copy\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-var-lib-kubelet\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xln\" (UniqueName: \"kubernetes.io/projected/34c5ccd1-df9d-4b19-b2ae-93d5827baef1-kube-api-access-s8xln\") pod \"node-ca-tgvsf\" (UID: \"34c5ccd1-df9d-4b19-b2ae-93d5827baef1\") " pod="openshift-image-registry/node-ca-tgvsf"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-kubelet\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-run-netns\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-log-socket\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-systemd\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.564212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-tuned\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.564764 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-var-lib-cni-multus\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.564764 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-conf-dir\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.564764 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.563910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-etc-openvswitch\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.564764 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f87b0945-3613-44a4-b60b-44deb2a1143c-iptables-alerter-script\") pod \"iptables-alerter-pckth\" (UID: \"f87b0945-3613-44a4-b60b-44deb2a1143c\") " pod="openshift-network-operator/iptables-alerter-pckth"
Sep 09 12:48:38.564764 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-node-log\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.564764 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-cnibin\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.564764 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-registration-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.564764 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-device-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.564764 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxw7r\" (UniqueName: \"kubernetes.io/projected/ffd649fb-0687-45dd-9acc-4a8d232f144d-kube-api-access-wxw7r\") pod \"node-resolver-qp9gk\" (UID: \"ffd649fb-0687-45dd-9acc-4a8d232f144d\") " pod="openshift-dns/node-resolver-qp9gk"
Sep 09 12:48:38.564764 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks7hp\" (UniqueName: \"kubernetes.io/projected/8dbba43e-056a-4448-85cd-36f295e6f710-kube-api-access-ks7hp\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-sysctl-d\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-sys\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-etc-selinux\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-sys-fs\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-system-cni-dir\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmmbz\" (UniqueName: \"kubernetes.io/projected/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-kube-api-access-cmmbz\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-var-lib-openvswitch\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.564994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f87b0945-3613-44a4-b60b-44deb2a1143c-host-slash\") pod \"iptables-alerter-pckth\" (UID: \"f87b0945-3613-44a4-b60b-44deb2a1143c\") " pod="openshift-network-operator/iptables-alerter-pckth"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34c5ccd1-df9d-4b19-b2ae-93d5827baef1-host\") pod \"node-ca-tgvsf\" (UID: \"34c5ccd1-df9d-4b19-b2ae-93d5827baef1\") " pod="openshift-image-registry/node-ca-tgvsf"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-kubernetes\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f87b0945-3613-44a4-b60b-44deb2a1143c-host-slash\") pod \"iptables-alerter-pckth\" (UID: \"f87b0945-3613-44a4-b60b-44deb2a1143c\") " pod="openshift-network-operator/iptables-alerter-pckth"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-run\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-host\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-cni-bin\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34c5ccd1-df9d-4b19-b2ae-93d5827baef1-host\") pod \"node-ca-tgvsf\" (UID: \"34c5ccd1-df9d-4b19-b2ae-93d5827baef1\") " pod="openshift-image-registry/node-ca-tgvsf"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-modprobe-d\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.565239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-var-lib-kubelet\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.565808 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.565808 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-systemd-units\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.565808 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-var-lib-cni-bin\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.565808 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgkkg\" (UniqueName: \"kubernetes.io/projected/611a1f11-6e3b-406c-97e3-5fbec3957485-kube-api-access-tgkkg\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.565808 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.565808 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.565308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-run-openvswitch\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.570591 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.570566    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 12:48:38.573938 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.573913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xln\" (UniqueName: \"kubernetes.io/projected/34c5ccd1-df9d-4b19-b2ae-93d5827baef1-kube-api-access-s8xln\") pod \"node-ca-tgvsf\" (UID: \"34c5ccd1-df9d-4b19-b2ae-93d5827baef1\") " pod="openshift-image-registry/node-ca-tgvsf"
Sep 09 12:48:38.574113 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.574095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sj97\" (UniqueName: \"kubernetes.io/projected/f87b0945-3613-44a4-b60b-44deb2a1143c-kube-api-access-8sj97\") pod \"iptables-alerter-pckth\" (UID: \"f87b0945-3613-44a4-b60b-44deb2a1143c\") " pod="openshift-network-operator/iptables-alerter-pckth"
Sep 09 12:48:38.574179 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.574095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxw7r\" (UniqueName: \"kubernetes.io/projected/ffd649fb-0687-45dd-9acc-4a8d232f144d-kube-api-access-wxw7r\") pod \"node-resolver-qp9gk\" (UID: \"ffd649fb-0687-45dd-9acc-4a8d232f144d\") " pod="openshift-dns/node-resolver-qp9gk"
Sep 09 12:48:38.622276 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:38.622241    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5785166f811993e15a031323bc2330c5.slice/crio-73d995ff266db60564ac43d66c8a707414c026f6a63ff47171cabaf3f7769b1f WatchSource:0}: Error finding container 73d995ff266db60564ac43d66c8a707414c026f6a63ff47171cabaf3f7769b1f: Status 404 returned error can't find the container with id 73d995ff266db60564ac43d66c8a707414c026f6a63ff47171cabaf3f7769b1f
Sep 09 12:48:38.624294 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:38.624267    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbae3b2574b8b7399924363fcb3393c.slice/crio-4ded78ed417ee5bd29c147560c0157d00e5b2278ed1a206570cc539f89116705 WatchSource:0}: Error finding container 4ded78ed417ee5bd29c147560c0157d00e5b2278ed1a206570cc539f89116705: Status 404 returned error can't find the container with id 4ded78ed417ee5bd29c147560c0157d00e5b2278ed1a206570cc539f89116705
Sep 09 12:48:38.628392 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.628377    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:48:38.665597 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28b841c2-6400-4ee4-bdf6-8f238cc60545-konnectivity-ca\") pod \"konnectivity-agent-qq7xl\" (UID: \"28b841c2-6400-4ee4-bdf6-8f238cc60545\") " pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:48:38.665597 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8dbba43e-056a-4448-85cd-36f295e6f710-ovn-node-metrics-cert\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.665876 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwlpb\" (UniqueName: \"kubernetes.io/projected/ac2e5020-766d-4573-afa9-8c3ec3ab350c-kube-api-access-fwlpb\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.665876 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-etc-kubernetes\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.665876 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-etc-kubernetes\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.665876 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-socket-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.665876 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:38.665876 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-run-netns\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.665876 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-socket-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.665876 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-slash\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.665876 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28b841c2-6400-4ee4-bdf6-8f238cc60545-agent-certs\") pod \"konnectivity-agent-qq7xl\" (UID: \"28b841c2-6400-4ee4-bdf6-8f238cc60545\") " pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8dbba43e-056a-4448-85cd-36f295e6f710-env-overrides\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-run-netns\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-slash\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.665961    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.665981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-system-cni-dir\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/611a1f11-6e3b-406c-97e3-5fbec3957485-cni-binary-copy\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.666048    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs podName:fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9 nodeName:}" failed. No retries permitted until 2026-09-09 12:48:39.166024083 +0000 UTC m=+2.085915403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs") pod "network-metrics-daemon-gjfxj" (UID: "fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-var-lib-kubelet\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-system-cni-dir\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-kubelet\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-run-netns\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-log-socket\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-systemd\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-kubelet\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-tuned\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28b841c2-6400-4ee4-bdf6-8f238cc60545-konnectivity-ca\") pod \"konnectivity-agent-qq7xl\" (UID: \"28b841c2-6400-4ee4-bdf6-8f238cc60545\") " pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-var-lib-cni-multus\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.666309 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-run-netns\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-var-lib-cni-multus\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-conf-dir\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-log-socket\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-etc-openvswitch\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-systemd\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-node-log\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-conf-dir\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-etc-openvswitch\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-cnibin\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-node-log\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-cnibin\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-registration-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-var-lib-kubelet\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-device-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks7hp\" (UniqueName: \"kubernetes.io/projected/8dbba43e-056a-4448-85cd-36f295e6f710-kube-api-access-ks7hp\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-registration-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-sysctl-d\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.667321 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-sys\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-device-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-etc-selinux\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-sys\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-sysctl-d\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-sys-fs\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-etc-selinux\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-system-cni-dir\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-sys-fs\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmmbz\" (UniqueName: \"kubernetes.io/projected/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-kube-api-access-cmmbz\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-system-cni-dir\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-var-lib-openvswitch\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-kubernetes\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8dbba43e-056a-4448-85cd-36f295e6f710-env-overrides\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-var-lib-openvswitch\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-kubernetes\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-run\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-host\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-cni-bin\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.666990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-modprobe-d\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-host\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667013    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-run\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-cni-bin\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-var-lib-kubelet\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-var-lib-kubelet\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-modprobe-d\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-systemd-units\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-var-lib-cni-bin\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-systemd-units\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgkkg\" (UniqueName: \"kubernetes.io/projected/611a1f11-6e3b-406c-97e3-5fbec3957485-kube-api-access-tgkkg\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-var-lib-cni-bin\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-run-openvswitch\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-sysconfig\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.668971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-socket-dir-parent\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-run-openvswitch\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-cni-netd\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59e0eeed-687d-47aa-8660-d3642cb43488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-cni-dir\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-sysconfig\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-cni-netd\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2014baf8-52ee-420c-89d0-205989b32b87-kubelet-config\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667403    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-socket-dir-parent\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-run-ovn\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-run-systemd\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2014baf8-52ee-420c-89d0-205989b32b87-kubelet-config\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8dbba43e-056a-4448-85cd-36f295e6f710-ovnkube-config\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-run-k8s-cni-cncf-io\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-run-ovn\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:38.669530 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-cni-dir\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6vhk\" (UniqueName: \"kubernetes.io/projected/59e0eeed-687d-47aa-8660-d3642cb43488-kube-api-access-w6vhk\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-run-k8s-cni-cncf-io\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-run-systemd\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8dbba43e-056a-4448-85cd-36f295e6f710-ovnkube-script-lib\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-sysctl-conf\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-lib-modules\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.667705    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-hostroot\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667739    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.667777    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret podName:2014baf8-52ee-420c-89d0-205989b32b87 nodeName:}" failed. No retries permitted until 2026-09-09 12:48:39.167761699 +0000 UTC m=+2.087653029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret") pod "global-pull-secret-syncer-7jcs4" (UID: "2014baf8-52ee-420c-89d0-205989b32b87") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-sysctl-conf\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-hostroot\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac2e5020-766d-4573-afa9-8c3ec3ab350c-lib-modules\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-daemon-config\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.667988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2014baf8-52ee-420c-89d0-205989b32b87-dbus\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:38.670017 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-cnibin\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8dbba43e-056a-4448-85cd-36f295e6f710-ovnkube-config\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-run-ovn-kubernetes\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-cnibin\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac2e5020-766d-4573-afa9-8c3ec3ab350c-tmp\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-run-multus-certs\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2014baf8-52ee-420c-89d0-205989b32b87-dbus\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njclh\" (UniqueName: \"kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh\") pod \"network-check-target-2n2wj\" (UID: \"ab6539a3-0367-425b-9dc0-cc894772df3a\") " pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dbba43e-056a-4448-85cd-36f295e6f710-host-run-ovn-kubernetes\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r44gn\" (UniqueName: \"kubernetes.io/projected/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-kube-api-access-r44gn\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/611a1f11-6e3b-406c-97e3-5fbec3957485-multus-daemon-config\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8dbba43e-056a-4448-85cd-36f295e6f710-ovnkube-script-lib\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/611a1f11-6e3b-406c-97e3-5fbec3957485-host-run-multus-certs\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.670472 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8dbba43e-056a-4448-85cd-36f295e6f710-ovn-node-metrics-cert\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.670962 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/611a1f11-6e3b-406c-97e3-5fbec3957485-cni-binary-copy\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.670962 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.670962 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.668989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.670962 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.669123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac2e5020-766d-4573-afa9-8c3ec3ab350c-etc-tuned\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.670962 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.669136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28b841c2-6400-4ee4-bdf6-8f238cc60545-agent-certs\") pod \"konnectivity-agent-qq7xl\" (UID: \"28b841c2-6400-4ee4-bdf6-8f238cc60545\") " pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:48:38.670962 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.670425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac2e5020-766d-4573-afa9-8c3ec3ab350c-tmp\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.674048 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.674024    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:48:38.674048 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.674050    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:48:38.674213 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.674064    2580 projected.go:196] Error preparing data for projected volume kube-api-access-njclh for pod openshift-network-diagnostics/network-check-target-2n2wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:38.674213 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:38.674126    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh podName:ab6539a3-0367-425b-9dc0-cc894772df3a nodeName:}" failed. No retries permitted until 2026-09-09 12:48:39.174110765 +0000 UTC m=+2.094002082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-njclh" (UniqueName: "kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh") pod "network-check-target-2n2wj" (UID: "ab6539a3-0367-425b-9dc0-cc894772df3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:38.674482 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.674461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwlpb\" (UniqueName: \"kubernetes.io/projected/ac2e5020-766d-4573-afa9-8c3ec3ab350c-kube-api-access-fwlpb\") pod \"tuned-vc5kq\" (UID: \"ac2e5020-766d-4573-afa9-8c3ec3ab350c\") " pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.675768 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.675744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmmbz\" (UniqueName: \"kubernetes.io/projected/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-kube-api-access-cmmbz\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:38.676140 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.676116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r44gn\" (UniqueName: \"kubernetes.io/projected/8f1a2660-8f7b-42e9-ae27-1f930df13a3c-kube-api-access-r44gn\") pod \"multus-additional-cni-plugins-bpb85\" (UID: \"8f1a2660-8f7b-42e9-ae27-1f930df13a3c\") " pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.676350 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.676322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks7hp\" (UniqueName: \"kubernetes.io/projected/8dbba43e-056a-4448-85cd-36f295e6f710-kube-api-access-ks7hp\") pod \"ovnkube-node-9cq52\" (UID: \"8dbba43e-056a-4448-85cd-36f295e6f710\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.676714 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.676693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgkkg\" (UniqueName: \"kubernetes.io/projected/611a1f11-6e3b-406c-97e3-5fbec3957485-kube-api-access-tgkkg\") pod \"multus-k82nw\" (UID: \"611a1f11-6e3b-406c-97e3-5fbec3957485\") " pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.676779 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.676702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6vhk\" (UniqueName: \"kubernetes.io/projected/59e0eeed-687d-47aa-8660-d3642cb43488-kube-api-access-w6vhk\") pod \"aws-ebs-csi-driver-node-llgnr\" (UID: \"59e0eeed-687d-47aa-8660-d3642cb43488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.784105 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.784014    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pckth"
Sep 09 12:48:38.793672 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:38.793649    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87b0945_3613_44a4_b60b_44deb2a1143c.slice/crio-7ee3598b754f2c05d9bee0bd45c94d9c47bc790f213ee0faaa41a1ffa94dce87 WatchSource:0}: Error finding container 7ee3598b754f2c05d9bee0bd45c94d9c47bc790f213ee0faaa41a1ffa94dce87: Status 404 returned error can't find the container with id 7ee3598b754f2c05d9bee0bd45c94d9c47bc790f213ee0faaa41a1ffa94dce87
Sep 09 12:48:38.804349 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.804329    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tgvsf"
Sep 09 12:48:38.811788 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:38.811768    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c5ccd1_df9d_4b19_b2ae_93d5827baef1.slice/crio-fc64386324be9bc0e8c98473b77b3bcd15b8b7f1801a3c3618ba160b17696d8a WatchSource:0}: Error finding container fc64386324be9bc0e8c98473b77b3bcd15b8b7f1801a3c3618ba160b17696d8a: Status 404 returned error can't find the container with id fc64386324be9bc0e8c98473b77b3bcd15b8b7f1801a3c3618ba160b17696d8a
Sep 09 12:48:38.821162 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.821143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qp9gk"
Sep 09 12:48:38.825800 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.825786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bpb85"
Sep 09 12:48:38.829278 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:38.829258    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd649fb_0687_45dd_9acc_4a8d232f144d.slice/crio-9e289b5362dcfe53742e01cd73c520e46b922fa142edd6b624d602804fcf84cc WatchSource:0}: Error finding container 9e289b5362dcfe53742e01cd73c520e46b922fa142edd6b624d602804fcf84cc: Status 404 returned error can't find the container with id 9e289b5362dcfe53742e01cd73c520e46b922fa142edd6b624d602804fcf84cc
Sep 09 12:48:38.834240 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:38.834211    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1a2660_8f7b_42e9_ae27_1f930df13a3c.slice/crio-4829bc851c5b580dda8fb0c5143bc391247184dfc2504c7006354cfc9abfb3de WatchSource:0}: Error finding container 4829bc851c5b580dda8fb0c5143bc391247184dfc2504c7006354cfc9abfb3de: Status 404 returned error can't find the container with id 4829bc851c5b580dda8fb0c5143bc391247184dfc2504c7006354cfc9abfb3de
Sep 09 12:48:38.845433 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.845414    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k82nw"
Sep 09 12:48:38.854093 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:38.854066    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611a1f11_6e3b_406c_97e3_5fbec3957485.slice/crio-b85557bdcfbd84c33d467f504238e1366d766c70ae1c98625b9da06ae1d5f976 WatchSource:0}: Error finding container b85557bdcfbd84c33d467f504238e1366d766c70ae1c98625b9da06ae1d5f976: Status 404 returned error can't find the container with id b85557bdcfbd84c33d467f504238e1366d766c70ae1c98625b9da06ae1d5f976
Sep 09 12:48:38.862637 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.862616    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:48:38.870314 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:38.870295    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b841c2_6400_4ee4_bdf6_8f238cc60545.slice/crio-60720488b22a1b81cb1c0b30fbd8540584795d4a347f70f1a73f6c94c9ca9bdb WatchSource:0}: Error finding container 60720488b22a1b81cb1c0b30fbd8540584795d4a347f70f1a73f6c94c9ca9bdb: Status 404 returned error can't find the container with id 60720488b22a1b81cb1c0b30fbd8540584795d4a347f70f1a73f6c94c9ca9bdb
Sep 09 12:48:38.881883 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.881864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:48:38.889646 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:38.889620    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbba43e_056a_4448_85cd_36f295e6f710.slice/crio-547d59fba4591ad6a962722c43e7111685d8d8e7048c63fa2c4c0b4ee83d85e3 WatchSource:0}: Error finding container 547d59fba4591ad6a962722c43e7111685d8d8e7048c63fa2c4c0b4ee83d85e3: Status 404 returned error can't find the container with id 547d59fba4591ad6a962722c43e7111685d8d8e7048c63fa2c4c0b4ee83d85e3
Sep 09 12:48:38.895578 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.895556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr"
Sep 09 12:48:38.899908 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:38.899889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vc5kq"
Sep 09 12:48:38.904557 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:38.904539    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e0eeed_687d_47aa_8660_d3642cb43488.slice/crio-3bff29a420d78277827912dec96a787df1ae2cfd1d8dcd9a031e89270906e4ba WatchSource:0}: Error finding container 3bff29a420d78277827912dec96a787df1ae2cfd1d8dcd9a031e89270906e4ba: Status 404 returned error can't find the container with id 3bff29a420d78277827912dec96a787df1ae2cfd1d8dcd9a031e89270906e4ba
Sep 09 12:48:38.908526 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:48:38.908508    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2e5020_766d_4573_afa9_8c3ec3ab350c.slice/crio-343e8c14cca93486b0473acfd78b36f09c3eb06bf55950a73e1dd5f333771a0e WatchSource:0}: Error finding container 343e8c14cca93486b0473acfd78b36f09c3eb06bf55950a73e1dd5f333771a0e: Status 404 returned error can't find the container with id 343e8c14cca93486b0473acfd78b36f09c3eb06bf55950a73e1dd5f333771a0e
Sep 09 12:48:39.171994 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.171910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:39.171994 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.171975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:39.172184 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:39.172106    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:39.172184 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:39.172158    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs podName:fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9 nodeName:}" failed. No retries permitted until 2026-09-09 12:48:40.172141939 +0000 UTC m=+3.092033275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs") pod "network-metrics-daemon-gjfxj" (UID: "fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:39.172563 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:39.172543    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:39.172662 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:39.172594    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret podName:2014baf8-52ee-420c-89d0-205989b32b87 nodeName:}" failed. No retries permitted until 2026-09-09 12:48:40.172580084 +0000 UTC m=+3.092471399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret") pod "global-pull-secret-syncer-7jcs4" (UID: "2014baf8-52ee-420c-89d0-205989b32b87") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:39.273483 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.273201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njclh\" (UniqueName: \"kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh\") pod \"network-check-target-2n2wj\" (UID: \"ab6539a3-0367-425b-9dc0-cc894772df3a\") " pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:39.273483 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:39.273382    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:48:39.273483 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:39.273398    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:48:39.273483 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:39.273410    2580 projected.go:196] Error preparing data for projected volume kube-api-access-njclh for pod openshift-network-diagnostics/network-check-target-2n2wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:39.273483 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:39.273458    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh podName:ab6539a3-0367-425b-9dc0-cc894772df3a nodeName:}" failed. No retries permitted until 2026-09-09 12:48:40.273442346 +0000 UTC m=+3.193333665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-njclh" (UniqueName: "kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh") pod "network-check-target-2n2wj" (UID: "ab6539a3-0367-425b-9dc0-cc894772df3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:39.344426 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.344396    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:48:39.356393 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.356366    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:48:39.505012 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.504907    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 12:43:38 +0000 UTC" deadline="2028-05-16 18:48:34.765004536 +0000 UTC"
Sep 09 12:48:39.505012 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.504966    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14765h59m55.260043177s"
Sep 09 12:48:39.591518 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.591462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-133.ec2.internal" event={"ID":"5785166f811993e15a031323bc2330c5","Type":"ContainerStarted","Data":"73d995ff266db60564ac43d66c8a707414c026f6a63ff47171cabaf3f7769b1f"}
Sep 09 12:48:39.593495 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.593466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal" event={"ID":"bcbae3b2574b8b7399924363fcb3393c","Type":"ContainerStarted","Data":"4ded78ed417ee5bd29c147560c0157d00e5b2278ed1a206570cc539f89116705"}
Sep 09 12:48:39.595557 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.595530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vc5kq" event={"ID":"ac2e5020-766d-4573-afa9-8c3ec3ab350c","Type":"ContainerStarted","Data":"343e8c14cca93486b0473acfd78b36f09c3eb06bf55950a73e1dd5f333771a0e"}
Sep 09 12:48:39.597394 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.597370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qq7xl" event={"ID":"28b841c2-6400-4ee4-bdf6-8f238cc60545","Type":"ContainerStarted","Data":"60720488b22a1b81cb1c0b30fbd8540584795d4a347f70f1a73f6c94c9ca9bdb"}
Sep 09 12:48:39.599099 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.599072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k82nw" event={"ID":"611a1f11-6e3b-406c-97e3-5fbec3957485","Type":"ContainerStarted","Data":"b85557bdcfbd84c33d467f504238e1366d766c70ae1c98625b9da06ae1d5f976"}
Sep 09 12:48:39.600755 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.600733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr" event={"ID":"59e0eeed-687d-47aa-8660-d3642cb43488","Type":"ContainerStarted","Data":"3bff29a420d78277827912dec96a787df1ae2cfd1d8dcd9a031e89270906e4ba"}
Sep 09 12:48:39.606161 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.606094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52" event={"ID":"8dbba43e-056a-4448-85cd-36f295e6f710","Type":"ContainerStarted","Data":"547d59fba4591ad6a962722c43e7111685d8d8e7048c63fa2c4c0b4ee83d85e3"}
Sep 09 12:48:39.610132 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.610077    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpb85" event={"ID":"8f1a2660-8f7b-42e9-ae27-1f930df13a3c","Type":"ContainerStarted","Data":"4829bc851c5b580dda8fb0c5143bc391247184dfc2504c7006354cfc9abfb3de"}
Sep 09 12:48:39.615434 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.615410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qp9gk" event={"ID":"ffd649fb-0687-45dd-9acc-4a8d232f144d","Type":"ContainerStarted","Data":"9e289b5362dcfe53742e01cd73c520e46b922fa142edd6b624d602804fcf84cc"}
Sep 09 12:48:39.617576 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.617546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tgvsf" event={"ID":"34c5ccd1-df9d-4b19-b2ae-93d5827baef1","Type":"ContainerStarted","Data":"fc64386324be9bc0e8c98473b77b3bcd15b8b7f1801a3c3618ba160b17696d8a"}
Sep 09 12:48:39.626458 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:39.626434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pckth" event={"ID":"f87b0945-3613-44a4-b60b-44deb2a1143c","Type":"ContainerStarted","Data":"7ee3598b754f2c05d9bee0bd45c94d9c47bc790f213ee0faaa41a1ffa94dce87"}
Sep 09 12:48:40.181853 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:40.181389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:40.181853 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:40.181467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:40.181853 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:40.181613    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:40.181853 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:40.181672    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs podName:fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9 nodeName:}" failed. No retries permitted until 2026-09-09 12:48:42.181658078 +0000 UTC m=+5.101549397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs") pod "network-metrics-daemon-gjfxj" (UID: "fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:40.182172 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:40.182056    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:40.182172 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:40.182104    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret podName:2014baf8-52ee-420c-89d0-205989b32b87 nodeName:}" failed. No retries permitted until 2026-09-09 12:48:42.182090627 +0000 UTC m=+5.101981948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret") pod "global-pull-secret-syncer-7jcs4" (UID: "2014baf8-52ee-420c-89d0-205989b32b87") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:40.282064 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:40.282026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njclh\" (UniqueName: \"kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh\") pod \"network-check-target-2n2wj\" (UID: \"ab6539a3-0367-425b-9dc0-cc894772df3a\") " pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:40.282273 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:40.282211    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:48:40.282273 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:40.282234    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:48:40.282273 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:40.282245    2580 projected.go:196] Error preparing data for projected volume kube-api-access-njclh for pod openshift-network-diagnostics/network-check-target-2n2wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:40.282447 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:40.282303    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh podName:ab6539a3-0367-425b-9dc0-cc894772df3a nodeName:}" failed. No retries permitted until 2026-09-09 12:48:42.282286541 +0000 UTC m=+5.202177864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-njclh" (UniqueName: "kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh") pod "network-check-target-2n2wj" (UID: "ab6539a3-0367-425b-9dc0-cc894772df3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:40.505867 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:40.505717    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 12:43:38 +0000 UTC" deadline="2028-06-03 19:54:40.991730248 +0000 UTC"
Sep 09 12:48:40.505867 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:40.505771    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15199h6m0.485963225s"
Sep 09 12:48:40.587137 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:40.586295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:40.587137 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:40.586436    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:48:40.587137 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:40.586859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:40.587137 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:40.586959    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:48:40.587137 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:40.587022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:40.587137 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:40.587087    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:48:40.953495 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:40.953413    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:48:41.178062 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:41.177999    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:48:42.201086 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:42.201046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:42.201571 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:42.201114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:42.201571 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:42.201263    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:42.201571 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:42.201340    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret podName:2014baf8-52ee-420c-89d0-205989b32b87 nodeName:}" failed. No retries permitted until 2026-09-09 12:48:46.201318668 +0000 UTC m=+9.121210000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret") pod "global-pull-secret-syncer-7jcs4" (UID: "2014baf8-52ee-420c-89d0-205989b32b87") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:42.201752 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:42.201733    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:42.201803 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:42.201790    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs podName:fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9 nodeName:}" failed. No retries permitted until 2026-09-09 12:48:46.201773393 +0000 UTC m=+9.121664730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs") pod "network-metrics-daemon-gjfxj" (UID: "fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:42.301637 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:42.301599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njclh\" (UniqueName: \"kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh\") pod \"network-check-target-2n2wj\" (UID: \"ab6539a3-0367-425b-9dc0-cc894772df3a\") " pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:42.301860 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:42.301761    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:48:42.301860 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:42.301783    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:48:42.301860 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:42.301794    2580 projected.go:196] Error preparing data for projected volume kube-api-access-njclh for pod openshift-network-diagnostics/network-check-target-2n2wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:42.301860 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:42.301867    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh podName:ab6539a3-0367-425b-9dc0-cc894772df3a nodeName:}" failed. No retries permitted until 2026-09-09 12:48:46.301848726 +0000 UTC m=+9.221740042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-njclh" (UniqueName: "kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh") pod "network-check-target-2n2wj" (UID: "ab6539a3-0367-425b-9dc0-cc894772df3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:42.585536 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:42.585456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:42.585704 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:42.585592    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:48:42.586006 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:42.585984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:42.586121 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:42.586086    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:48:42.586182 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:42.586135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:42.586238 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:42.586202    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:48:44.585572 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:44.585541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:44.586030 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:44.585642    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:48:44.586030 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:44.585934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:44.586130 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:44.586050    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:48:44.586130 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:44.586081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:44.586219 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:44.586167    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:48:46.235873 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:46.235813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:46.236340 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:46.235915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:46.236340 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:46.236014    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:46.236340 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:46.236065    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs podName:fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9 nodeName:}" failed. No retries permitted until 2026-09-09 12:48:54.236050031 +0000 UTC m=+17.155941348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs") pod "network-metrics-daemon-gjfxj" (UID: "fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:46.236340 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:46.236105    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:46.236340 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:46.236125    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret podName:2014baf8-52ee-420c-89d0-205989b32b87 nodeName:}" failed. No retries permitted until 2026-09-09 12:48:54.236119217 +0000 UTC m=+17.156010534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret") pod "global-pull-secret-syncer-7jcs4" (UID: "2014baf8-52ee-420c-89d0-205989b32b87") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:46.336994 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:46.336912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njclh\" (UniqueName: \"kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh\") pod \"network-check-target-2n2wj\" (UID: \"ab6539a3-0367-425b-9dc0-cc894772df3a\") " pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:46.337162 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:46.337097    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:48:46.337162 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:46.337119    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:48:46.337162 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:46.337132    2580 projected.go:196] Error preparing data for projected volume kube-api-access-njclh for pod openshift-network-diagnostics/network-check-target-2n2wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:46.337332 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:46.337201    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh podName:ab6539a3-0367-425b-9dc0-cc894772df3a nodeName:}" failed. No retries permitted until 2026-09-09 12:48:54.337182423 +0000 UTC m=+17.257073751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-njclh" (UniqueName: "kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh") pod "network-check-target-2n2wj" (UID: "ab6539a3-0367-425b-9dc0-cc894772df3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:46.586178 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:46.586147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:46.586341 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:46.586270    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:48:46.586672 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:46.586647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:46.586777 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:46.586761    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:48:46.586869 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:46.586812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:46.586923 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:46.586906    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:48:48.586188 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:48.586150    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:48.586628 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:48.586281    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:48:48.586747 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:48.586728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:48.586877 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:48.586859    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:48:48.586956 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:48.586933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:48.587085 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:48.587057    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:48:50.586245 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:50.586206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:50.586245 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:50.586235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:50.586804 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:50.586364    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:48:50.586804 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:50.586417    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:50.586804 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:50.586472    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:48:50.586969 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:50.586885    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:48:52.586332 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:52.586289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:52.586332 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:52.586323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:52.586800 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:52.586351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:52.586800 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:52.586421    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:48:52.586800 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:52.586480    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:48:52.586800 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:52.586560    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:48:54.294223 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:54.294190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:54.294747 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:54.294250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:54.294747 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:54.294353    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:54.294747 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:54.294361    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:54.294747 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:54.294412    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs podName:fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:10.294392207 +0000 UTC m=+33.214283524 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs") pod "network-metrics-daemon-gjfxj" (UID: "fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:48:54.294747 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:54.294429    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret podName:2014baf8-52ee-420c-89d0-205989b32b87 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:10.294422233 +0000 UTC m=+33.214313549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret") pod "global-pull-secret-syncer-7jcs4" (UID: "2014baf8-52ee-420c-89d0-205989b32b87") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:48:54.394836 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:54.394785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njclh\" (UniqueName: \"kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh\") pod \"network-check-target-2n2wj\" (UID: \"ab6539a3-0367-425b-9dc0-cc894772df3a\") " pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:54.395027 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:54.394968    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:48:54.395027 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:54.394994    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:48:54.395027 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:54.395004    2580 projected.go:196] Error preparing data for projected volume kube-api-access-njclh for pod openshift-network-diagnostics/network-check-target-2n2wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:54.395183 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:54.395055    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh podName:ab6539a3-0367-425b-9dc0-cc894772df3a nodeName:}" failed. No retries permitted until 2026-09-09 12:49:10.395042067 +0000 UTC m=+33.314933384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-njclh" (UniqueName: "kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh") pod "network-check-target-2n2wj" (UID: "ab6539a3-0367-425b-9dc0-cc894772df3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:48:54.586223 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:54.586138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:54.586223 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:54.586169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:54.586223 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:54.586169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:54.586494 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:54.586265    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:48:54.586494 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:54.586365    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:48:54.586494 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:54.586458    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:48:56.585786 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:56.585740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:56.586281 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:56.585900    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:48:56.586281 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:56.585929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:56.586281 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:56.585952    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:56.586281 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:56.586047    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:48:56.586281 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:56.586133    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:48:57.666814 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:57.666594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-133.ec2.internal" event={"ID":"5785166f811993e15a031323bc2330c5","Type":"ContainerStarted","Data":"78957b1fd27e2882799bc66b22fffb07c0518b4964db595b693d368ce5f18ffc"}
Sep 09 12:48:57.668359 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:57.668336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vc5kq" event={"ID":"ac2e5020-766d-4573-afa9-8c3ec3ab350c","Type":"ContainerStarted","Data":"4e9951fbe984c11e86bcdc9a7bcee9cdcafebb5375fde42e1e9c1e9ac0ac76dc"}
Sep 09 12:48:57.670064 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:57.670042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52" event={"ID":"8dbba43e-056a-4448-85cd-36f295e6f710","Type":"ContainerStarted","Data":"8bf55c9fd272e3649bffeee9cc8816c22d9ba28d46662f6adbc864d3ce4cf998"}
Sep 09 12:48:57.680284 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:57.680239    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-133.ec2.internal" podStartSLOduration=19.680225042 podStartE2EDuration="19.680225042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:48:57.680157844 +0000 UTC m=+20.600049182" watchObservedRunningTime="2026-09-09 12:48:57.680225042 +0000 UTC m=+20.600116382"
Sep 09 12:48:57.700449 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:57.700404    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vc5kq" podStartSLOduration=2.245027015 podStartE2EDuration="20.700389103s" totalImagesPullingTime="18.455362088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:48:57.7002617 +0000 UTC m=+20.620153039" watchObservedRunningTime="2026-09-09 12:48:57.700389103 +0000 UTC m=+20.620280441"
Sep 09 12:48:58.586497 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.586191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:48:58.586648 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.586289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:48:58.586648 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:58.586548    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:48:58.586648 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.586300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:48:58.586648 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:58.586624    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:48:58.586807 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:48:58.586664    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:48:58.675045 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.675005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr" event={"ID":"59e0eeed-687d-47aa-8660-d3642cb43488","Type":"ContainerStarted","Data":"8a493fea75bc55dee10d5bc541309dc5e097cb1714a99f3b4dcbfbcb07d7a3f9"}
Sep 09 12:48:58.679345 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.679286    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 12:48:58.679678 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.679657    2580 generic.go:356] "Generic (PLEG): container finished" podID="8dbba43e-056a-4448-85cd-36f295e6f710" containerID="746535e6d14d3e1a9e187f61076355800de642df286762012c2dea562dc29a21" exitCode=1
Sep 09 12:48:58.679791 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.679725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52" event={"ID":"8dbba43e-056a-4448-85cd-36f295e6f710","Type":"ContainerStarted","Data":"0970d09f705bbbe85ea5fc9500b7f8834b5403fee469951f3c47ee16e34cb5e7"}
Sep 09 12:48:58.679867 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.679798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52" event={"ID":"8dbba43e-056a-4448-85cd-36f295e6f710","Type":"ContainerStarted","Data":"6ab021ef3f2290da1c0e8d69e3561a23997df62a5fcfc3b50ba1ae47ac7e85ee"}
Sep 09 12:48:58.679917 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.679868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52" event={"ID":"8dbba43e-056a-4448-85cd-36f295e6f710","Type":"ContainerStarted","Data":"c564cd24c46e94a1a704ad68a571e07a3d8fe554a6e6ca6b59bad9908eda9ce6"}
Sep 09 12:48:58.679917 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.679906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52" event={"ID":"8dbba43e-056a-4448-85cd-36f295e6f710","Type":"ContainerStarted","Data":"b8e4fa6471895f78b2385a41a36a80518f9b6fc998151fa620a91d56fe7777d5"}
Sep 09 12:48:58.680022 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.679937    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52" event={"ID":"8dbba43e-056a-4448-85cd-36f295e6f710","Type":"ContainerDied","Data":"746535e6d14d3e1a9e187f61076355800de642df286762012c2dea562dc29a21"}
Sep 09 12:48:58.681574 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.681548    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f1a2660-8f7b-42e9-ae27-1f930df13a3c" containerID="00924643d3727a41e9d06ecd7ecbb8844546cb49e0e781f3892ffa6f0bd9faaf" exitCode=0
Sep 09 12:48:58.681674 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.681633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpb85" event={"ID":"8f1a2660-8f7b-42e9-ae27-1f930df13a3c","Type":"ContainerDied","Data":"00924643d3727a41e9d06ecd7ecbb8844546cb49e0e781f3892ffa6f0bd9faaf"}
Sep 09 12:48:58.683542 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.683517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qp9gk" event={"ID":"ffd649fb-0687-45dd-9acc-4a8d232f144d","Type":"ContainerStarted","Data":"360093342d4b255e339f48ce4bf64d24a7d39a4c355a192aa581536e5516fe79"}
Sep 09 12:48:58.685431 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.685403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tgvsf" event={"ID":"34c5ccd1-df9d-4b19-b2ae-93d5827baef1","Type":"ContainerStarted","Data":"379a013d888690c0ad3775574b46e455294edb94fbf266c674ea3dd0555062de"}
Sep 09 12:48:58.687564 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.687277    2580 generic.go:356] "Generic (PLEG): container finished" podID="bcbae3b2574b8b7399924363fcb3393c" containerID="cb9cb883ca610ee21fcfe306c6e00b11f92240a14352d35ac5c2b1891fb9dac8" exitCode=0
Sep 09 12:48:58.687564 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.687363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal" event={"ID":"bcbae3b2574b8b7399924363fcb3393c","Type":"ContainerDied","Data":"cb9cb883ca610ee21fcfe306c6e00b11f92240a14352d35ac5c2b1891fb9dac8"}
Sep 09 12:48:58.689560 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.689535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qq7xl" event={"ID":"28b841c2-6400-4ee4-bdf6-8f238cc60545","Type":"ContainerStarted","Data":"3853c19734fd262233bb78e6c7b6aa7c6466ab11da244ed60daa835610807dfc"}
Sep 09 12:48:58.692003 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.691978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k82nw" event={"ID":"611a1f11-6e3b-406c-97e3-5fbec3957485","Type":"ContainerStarted","Data":"bba3eacc85abe588e976c9df23ed265ea5c0b2c2f6c31d6b1d1831b2ce4c9d67"}
Sep 09 12:48:58.718753 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.718491    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qq7xl" podStartSLOduration=3.49180685 podStartE2EDuration="21.718471418s" totalImagesPullingTime="18.226664568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:48:58.718076801 +0000 UTC m=+21.637968142" watchObservedRunningTime="2026-09-09 12:48:58.718471418 +0000 UTC m=+21.638362758"
Sep 09 12:48:58.767605 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.766980    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tgvsf" podStartSLOduration=6.879103826 podStartE2EDuration="21.766957748s" totalImagesPullingTime="14.887853922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:48:58.750715582 +0000 UTC m=+21.670606926" watchObservedRunningTime="2026-09-09 12:48:58.766957748 +0000 UTC m=+21.686849066"
Sep 09 12:48:58.767605 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.767377    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k82nw" podStartSLOduration=2.848909563 podStartE2EDuration="21.767367778s" totalImagesPullingTime="18.918458215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:48:58.766916495 +0000 UTC m=+21.686807836" watchObservedRunningTime="2026-09-09 12:48:58.767367778 +0000 UTC m=+21.687259118"
Sep 09 12:48:58.782382 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:58.782336    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qp9gk" podStartSLOduration=6.911980041 podStartE2EDuration="21.78232267s" totalImagesPullingTime="14.870342629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:48:58.782148498 +0000 UTC m=+21.702039840" watchObservedRunningTime="2026-09-09 12:48:58.78232267 +0000 UTC m=+21.702214008"
Sep 09 12:48:59.020844 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:59.020802    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 12:48:59.540672 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:59.540561    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T12:48:59.020832489Z","UUID":"5e3c2477-b841-41ad-827c-2d41d6e97beb","Handler":null,"Name":"","Endpoint":""}
Sep 09 12:48:59.542352 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:59.542323    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 12:48:59.542489 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:59.542365    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 12:48:59.697186 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:59.697135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr" event={"ID":"59e0eeed-687d-47aa-8660-d3642cb43488","Type":"ContainerStarted","Data":"1bf131652dc3bc7d40fd688e5336e3fac998762312c8a7687a254ae586292854"}
Sep 09 12:48:59.699690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:59.699311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pckth" event={"ID":"f87b0945-3613-44a4-b60b-44deb2a1143c","Type":"ContainerStarted","Data":"6d1f11a189ebf59401b8bd06e5b286e4f9c0b81973102aa29f30a8e160306c3d"}
Sep 09 12:48:59.701587 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:59.701556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal" event={"ID":"bcbae3b2574b8b7399924363fcb3393c","Type":"ContainerStarted","Data":"46578d981f8699c662ca729aebaf7361a63171b7f18655d42fd68b0f4d38949e"}
Sep 09 12:48:59.713860 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:59.713793    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pckth" podStartSLOduration=4.145566177 podStartE2EDuration="22.713774521s" totalImagesPullingTime="18.568208344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:48:59.713695714 +0000 UTC m=+22.633587056" watchObservedRunningTime="2026-09-09 12:48:59.713774521 +0000 UTC m=+22.633665862"
Sep 09 12:48:59.727913 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:48:59.727868    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-133.ec2.internal" podStartSLOduration=21.727849796 podStartE2EDuration="21.727849796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:48:59.727407315 +0000 UTC m=+22.647298654" watchObservedRunningTime="2026-09-09 12:48:59.727849796 +0000 UTC m=+22.647741137"
Sep 09 12:49:00.586299 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:00.586256    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:49:00.586299 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:00.586287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:49:00.586536 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:00.586271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:00.586536 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:00.586394    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:49:00.586536 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:00.586512    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:49:00.586659 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:00.586589    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:49:00.706838 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:00.706777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr" event={"ID":"59e0eeed-687d-47aa-8660-d3642cb43488","Type":"ContainerStarted","Data":"39e5c5311f2918279d34b2f0b21ee64f2528157d8945d94a3c934dce1fde3a1d"}
Sep 09 12:49:00.710505 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:00.710483    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 12:49:00.710998 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:00.710973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52" event={"ID":"8dbba43e-056a-4448-85cd-36f295e6f710","Type":"ContainerStarted","Data":"434f218d9b9061bd0bbe0a24ac452b91a03011800b9da133f9caf17ca753816e"}
Sep 09 12:49:00.726067 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:00.725996    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-llgnr" podStartSLOduration=3.5672206429999997 podStartE2EDuration="23.725977435s" totalImagesPullingTime="20.158756792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:00.725586373 +0000 UTC m=+23.645477712" watchObservedRunningTime="2026-09-09 12:49:00.725977435 +0000 UTC m=+23.645868775"
Sep 09 12:49:01.806026 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:01.805972    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:49:01.807028 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:01.806996    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:49:02.586185 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:02.586106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:49:02.586185 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:02.586131    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:02.586185 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:02.586131    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:49:02.586448 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:02.586241    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:49:02.586448 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:02.586315    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:49:02.586448 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:02.586379    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:49:02.715780 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:02.715745    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:49:02.716757 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:02.716736    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qq7xl"
Sep 09 12:49:03.724090 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:03.723861    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 12:49:03.724845 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:03.724376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52" event={"ID":"8dbba43e-056a-4448-85cd-36f295e6f710","Type":"ContainerStarted","Data":"5e8f34170b0734d7dde6f979714e94bfd4f62fe6c516d3e346705d1e2161a7d7"}
Sep 09 12:49:03.724845 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:03.724746    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:49:03.724951 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:03.724924    2580 scope.go:117] "RemoveContainer" containerID="746535e6d14d3e1a9e187f61076355800de642df286762012c2dea562dc29a21"
Sep 09 12:49:03.726839 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:03.726790    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f1a2660-8f7b-42e9-ae27-1f930df13a3c" containerID="49003b35280121fcc3d98810f98655f36817c09b51349059c1ab5112a4d999fb" exitCode=0
Sep 09 12:49:03.726952 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:03.726856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpb85" event={"ID":"8f1a2660-8f7b-42e9-ae27-1f930df13a3c","Type":"ContainerDied","Data":"49003b35280121fcc3d98810f98655f36817c09b51349059c1ab5112a4d999fb"}
Sep 09 12:49:03.742609 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:03.742589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:49:04.585331 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:04.585265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:04.585470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:04.585390    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:49:04.585470 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:04.585393    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:49:04.585470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:04.585428    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:49:04.585632 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:04.585537    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:49:04.585676 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:04.585634    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:49:04.734231 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:04.734050    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 12:49:04.734681 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:04.734624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52" event={"ID":"8dbba43e-056a-4448-85cd-36f295e6f710","Type":"ContainerStarted","Data":"98d683de223110b2cede513c6a5e8fd370ad6747764d99f64dc7113a1c4cfeed"}
Sep 09 12:49:04.734963 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:04.734940    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:49:04.735110 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:04.735011    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:49:04.737001 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:04.736973    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f1a2660-8f7b-42e9-ae27-1f930df13a3c" containerID="827b4ee7102d7e0d171d08f6d531362e95021d1977d86cc82fafb03142d9873a" exitCode=0
Sep 09 12:49:04.737125 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:04.737086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpb85" event={"ID":"8f1a2660-8f7b-42e9-ae27-1f930df13a3c","Type":"ContainerDied","Data":"827b4ee7102d7e0d171d08f6d531362e95021d1977d86cc82fafb03142d9873a"}
Sep 09 12:49:04.751844 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:04.751790    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:49:04.763097 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:04.763049    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52" podStartSLOduration=9.252653836 podStartE2EDuration="27.763032806s" totalImagesPullingTime="18.51037897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:04.762643548 +0000 UTC m=+27.682534893" watchObservedRunningTime="2026-09-09 12:49:04.763032806 +0000 UTC m=+27.682924144"
Sep 09 12:49:05.072034 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:05.071988    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7jcs4"]
Sep 09 12:49:05.072260 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:05.072160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:49:05.072326 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:05.072254    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:49:05.072644 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:05.072617    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2n2wj"]
Sep 09 12:49:05.072763 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:05.072723    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:05.072864 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:05.072835    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:49:05.075146 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:05.075122    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gjfxj"]
Sep 09 12:49:05.075241 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:05.075230    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:49:05.075339 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:05.075313    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:49:05.741932 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:05.741891    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f1a2660-8f7b-42e9-ae27-1f930df13a3c" containerID="ea6d6e5b68533b2034c1ae3cc4603ed22b8ca74c312b4be34b61c9555d224488" exitCode=0
Sep 09 12:49:05.742297 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:05.741975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpb85" event={"ID":"8f1a2660-8f7b-42e9-ae27-1f930df13a3c","Type":"ContainerDied","Data":"ea6d6e5b68533b2034c1ae3cc4603ed22b8ca74c312b4be34b61c9555d224488"}
Sep 09 12:49:06.586069 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:06.585979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:49:06.586069 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:06.586030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:49:06.586069 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:06.586030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:06.586285 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:06.586098    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:49:06.586285 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:06.586206    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:49:06.586285 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:06.586274    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:49:08.586310 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:08.586232    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:08.586940 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:08.586356    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2n2wj" podUID="ab6539a3-0367-425b-9dc0-cc894772df3a"
Sep 09 12:49:08.586940 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:08.586233    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:49:08.586940 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:08.586445    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjfxj" podUID="fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9"
Sep 09 12:49:08.586940 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:08.586233    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:49:08.586940 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:08.586504    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7jcs4" podUID="2014baf8-52ee-420c-89d0-205989b32b87"
Sep 09 12:49:09.353963 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.353686    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-133.ec2.internal" event="NodeReady"
Sep 09 12:49:09.354130 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.354091    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 12:49:09.397387 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.397346    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vhfsf"]
Sep 09 12:49:09.450781 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.450744    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p8d9s"]
Sep 09 12:49:09.450960 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.450914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.453698 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.453660    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 12:49:09.453698 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.453695    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 12:49:09.453900 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.453836    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pmjpj\""
Sep 09 12:49:09.466080 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.465588    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p8d9s"]
Sep 09 12:49:09.466080 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.465634    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vhfsf"]
Sep 09 12:49:09.466080 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.465690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p8d9s"
Sep 09 12:49:09.466080 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.465751    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gtq25"]
Sep 09 12:49:09.468611 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.468580    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 12:49:09.468755 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.468633    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 12:49:09.468755 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.468585    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 12:49:09.468918 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.468777    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4hv86\""
Sep 09 12:49:09.485054 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.485028    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gtq25"]
Sep 09 12:49:09.485160 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.485139    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.487919 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.487851    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 12:49:09.488035 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.488016    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 12:49:09.488091 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.488030    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 12:49:09.488091 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.488086    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 12:49:09.488170 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.488018    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s29r7\""
Sep 09 12:49:09.511305 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.511284    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qp9gk_ffd649fb-0687-45dd-9acc-4a8d232f144d/dns-node-resolver/0.log"
Sep 09 12:49:09.605736 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.605708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl8hk\" (UniqueName: \"kubernetes.io/projected/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-kube-api-access-bl8hk\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.605736 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.605740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwwqk\" (UniqueName: \"kubernetes.io/projected/7faebb79-50d6-4521-b829-ba8c91fb8a9b-kube-api-access-mwwqk\") pod \"ingress-canary-p8d9s\" (UID: \"7faebb79-50d6-4521-b829-ba8c91fb8a9b\") " pod="openshift-ingress-canary/ingress-canary-p8d9s"
Sep 09 12:49:09.606216 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.605857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.606216 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.605905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/739e03ea-ddf4-4d97-9552-851e87b324da-crio-socket\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.606216 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.605943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/739e03ea-ddf4-4d97-9552-851e87b324da-data-volume\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.606216 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.605960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.606216 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.605977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-config-volume\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.606216 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.606053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhqzd\" (UniqueName: \"kubernetes.io/projected/739e03ea-ddf4-4d97-9552-851e87b324da-kube-api-access-nhqzd\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.606216 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.606090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.606216 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.606113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-tmp-dir\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.606216 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.606136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7faebb79-50d6-4521-b829-ba8c91fb8a9b-cert\") pod \"ingress-canary-p8d9s\" (UID: \"7faebb79-50d6-4521-b829-ba8c91fb8a9b\") " pod="openshift-ingress-canary/ingress-canary-p8d9s"
Sep 09 12:49:09.706931 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.706890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.707125 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.706944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/739e03ea-ddf4-4d97-9552-851e87b324da-crio-socket\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.707125 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.706979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/739e03ea-ddf4-4d97-9552-851e87b324da-data-volume\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.707125 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.706998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.707125 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.707018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-config-volume\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.707125 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.707048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhqzd\" (UniqueName: \"kubernetes.io/projected/739e03ea-ddf4-4d97-9552-851e87b324da-kube-api-access-nhqzd\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.707125 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:09.707069    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:49:09.707125 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:09.707114    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:49:09.707452 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:09.707138    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:49:09.707452 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:09.707150    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls podName:c3a0dfbf-070e-4c73-afd5-be6a620d8bb2 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:10.207128135 +0000 UTC m=+33.127019456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls") pod "dns-default-vhfsf" (UID: "c3a0dfbf-070e-4c73-afd5-be6a620d8bb2") : secret "dns-default-metrics-tls" not found
Sep 09 12:49:09.707452 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:09.707169    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls podName:739e03ea-ddf4-4d97-9552-851e87b324da nodeName:}" failed. No retries permitted until 2026-09-09 12:49:10.207157922 +0000 UTC m=+33.127049259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gtq25" (UID: "739e03ea-ddf4-4d97-9552-851e87b324da") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:49:09.707452 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.707076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.707452 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.707202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-tmp-dir\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.707452 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:09.707211    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm podName:739e03ea-ddf4-4d97-9552-851e87b324da nodeName:}" failed. No retries permitted until 2026-09-09 12:49:10.207194223 +0000 UTC m=+33.127085560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gtq25" (UID: "739e03ea-ddf4-4d97-9552-851e87b324da") : configmap "kube-rbac-proxy" not found
Sep 09 12:49:09.707452 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.707238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7faebb79-50d6-4521-b829-ba8c91fb8a9b-cert\") pod \"ingress-canary-p8d9s\" (UID: \"7faebb79-50d6-4521-b829-ba8c91fb8a9b\") " pod="openshift-ingress-canary/ingress-canary-p8d9s"
Sep 09 12:49:09.707452 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.707263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl8hk\" (UniqueName: \"kubernetes.io/projected/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-kube-api-access-bl8hk\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.707452 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.707288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwwqk\" (UniqueName: \"kubernetes.io/projected/7faebb79-50d6-4521-b829-ba8c91fb8a9b-kube-api-access-mwwqk\") pod \"ingress-canary-p8d9s\" (UID: \"7faebb79-50d6-4521-b829-ba8c91fb8a9b\") " pod="openshift-ingress-canary/ingress-canary-p8d9s"
Sep 09 12:49:09.707452 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.707316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/739e03ea-ddf4-4d97-9552-851e87b324da-crio-socket\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.707452 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.707402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/739e03ea-ddf4-4d97-9552-851e87b324da-data-volume\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.708014 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.707518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-tmp-dir\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.708014 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.707786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-config-volume\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.712045 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.712024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7faebb79-50d6-4521-b829-ba8c91fb8a9b-cert\") pod \"ingress-canary-p8d9s\" (UID: \"7faebb79-50d6-4521-b829-ba8c91fb8a9b\") " pod="openshift-ingress-canary/ingress-canary-p8d9s"
Sep 09 12:49:09.716440 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.716416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl8hk\" (UniqueName: \"kubernetes.io/projected/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-kube-api-access-bl8hk\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:09.716610 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.716590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhqzd\" (UniqueName: \"kubernetes.io/projected/739e03ea-ddf4-4d97-9552-851e87b324da-kube-api-access-nhqzd\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:09.717254 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.717235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwwqk\" (UniqueName: \"kubernetes.io/projected/7faebb79-50d6-4521-b829-ba8c91fb8a9b-kube-api-access-mwwqk\") pod \"ingress-canary-p8d9s\" (UID: \"7faebb79-50d6-4521-b829-ba8c91fb8a9b\") " pod="openshift-ingress-canary/ingress-canary-p8d9s"
Sep 09 12:49:09.776135 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:09.776040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p8d9s"
Sep 09 12:49:10.211604 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.211562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:10.211754 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.211617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:10.211754 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.211710    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:49:10.211754 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.211718    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:49:10.211754 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.211739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:10.211930 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.211760    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls podName:739e03ea-ddf4-4d97-9552-851e87b324da nodeName:}" failed. No retries permitted until 2026-09-09 12:49:11.21174693 +0000 UTC m=+34.131638247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gtq25" (UID: "739e03ea-ddf4-4d97-9552-851e87b324da") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:49:10.211930 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.211775    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls podName:c3a0dfbf-070e-4c73-afd5-be6a620d8bb2 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:11.211769425 +0000 UTC m=+34.131660742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls") pod "dns-default-vhfsf" (UID: "c3a0dfbf-070e-4c73-afd5-be6a620d8bb2") : secret "dns-default-metrics-tls" not found
Sep 09 12:49:10.211930 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.211792    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:49:10.211930 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.211858    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm podName:739e03ea-ddf4-4d97-9552-851e87b324da nodeName:}" failed. No retries permitted until 2026-09-09 12:49:11.21184017 +0000 UTC m=+34.131731505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gtq25" (UID: "739e03ea-ddf4-4d97-9552-851e87b324da") : configmap "kube-rbac-proxy" not found
Sep 09 12:49:10.312700 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.312615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:49:10.312700 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.312668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:49:10.312934 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.312759    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:49:10.312934 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.312778    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:49:10.312934 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.312803    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs podName:fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:42.312791197 +0000 UTC m=+65.232682513 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs") pod "network-metrics-daemon-gjfxj" (UID: "fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:49:10.312934 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.312862    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret podName:2014baf8-52ee-420c-89d0-205989b32b87 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:42.312846381 +0000 UTC m=+65.232737712 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret") pod "global-pull-secret-syncer-7jcs4" (UID: "2014baf8-52ee-420c-89d0-205989b32b87") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:49:10.319144 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.319090    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p8d9s"]
Sep 09 12:49:10.354124 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:10.354081    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7faebb79_50d6_4521_b829_ba8c91fb8a9b.slice/crio-d916862287aad9a999976f04fab38b373decdb67fc10fce8cf4ba6883e2fa5ce WatchSource:0}: Error finding container d916862287aad9a999976f04fab38b373decdb67fc10fce8cf4ba6883e2fa5ce: Status 404 returned error can't find the container with id d916862287aad9a999976f04fab38b373decdb67fc10fce8cf4ba6883e2fa5ce
Sep 09 12:49:10.413287 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.413259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njclh\" (UniqueName: \"kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh\") pod \"network-check-target-2n2wj\" (UID: \"ab6539a3-0367-425b-9dc0-cc894772df3a\") " pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:10.413428 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.413418    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:49:10.413469 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.413435    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:49:10.413469 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.413444    2580 projected.go:196] Error preparing data for projected volume kube-api-access-njclh for pod openshift-network-diagnostics/network-check-target-2n2wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:49:10.413530 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:10.413497    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh podName:ab6539a3-0367-425b-9dc0-cc894772df3a nodeName:}" failed. No retries permitted until 2026-09-09 12:49:42.41348191 +0000 UTC m=+65.333373237 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-njclh" (UniqueName: "kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh") pod "network-check-target-2n2wj" (UID: "ab6539a3-0367-425b-9dc0-cc894772df3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:49:10.481011 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.480989    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tgvsf_34c5ccd1-df9d-4b19-b2ae-93d5827baef1/node-ca/0.log"
Sep 09 12:49:10.585616 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.585534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:49:10.585616 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.585569    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:10.585616 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.585581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:49:10.588353 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.588329    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 12:49:10.588467 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.588376    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ns67g\""
Sep 09 12:49:10.589354 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.589325    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 12:49:10.589475 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.589368    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qcx5j\""
Sep 09 12:49:10.589475 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.589370    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 12:49:10.589475 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.589430    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 12:49:10.759746 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.759710    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f1a2660-8f7b-42e9-ae27-1f930df13a3c" containerID="119fa13d17026bd5e6ae393bce7161e3f60abb213f11a72bd599f9fdf17c361a" exitCode=0
Sep 09 12:49:10.760565 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.759791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpb85" event={"ID":"8f1a2660-8f7b-42e9-ae27-1f930df13a3c","Type":"ContainerDied","Data":"119fa13d17026bd5e6ae393bce7161e3f60abb213f11a72bd599f9fdf17c361a"}
Sep 09 12:49:10.761102 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.761078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p8d9s" event={"ID":"7faebb79-50d6-4521-b829-ba8c91fb8a9b","Type":"ContainerStarted","Data":"d916862287aad9a999976f04fab38b373decdb67fc10fce8cf4ba6883e2fa5ce"}
Sep 09 12:49:10.771702 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.771683    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-6tqqj"]
Sep 09 12:49:10.789735 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.789713    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-6tqqj"]
Sep 09 12:49:10.789853 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.789801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-6tqqj"
Sep 09 12:49:10.792411 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.792390    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 12:49:10.792522 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.792425    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 12:49:10.792522 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.792449    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 12:49:10.792522 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.792501    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rwd9v\""
Sep 09 12:49:10.792522 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.792457    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 12:49:10.917647 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.917387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d6941c31-81b1-45b3-a568-363abcc7b706-signing-key\") pod \"service-ca-75d99c484-6tqqj\" (UID: \"d6941c31-81b1-45b3-a568-363abcc7b706\") " pod="openshift-service-ca/service-ca-75d99c484-6tqqj"
Sep 09 12:49:10.917805 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.917668    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzktp\" (UniqueName: \"kubernetes.io/projected/d6941c31-81b1-45b3-a568-363abcc7b706-kube-api-access-xzktp\") pod \"service-ca-75d99c484-6tqqj\" (UID: \"d6941c31-81b1-45b3-a568-363abcc7b706\") " pod="openshift-service-ca/service-ca-75d99c484-6tqqj"
Sep 09 12:49:10.917805 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:10.917698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d6941c31-81b1-45b3-a568-363abcc7b706-signing-cabundle\") pod \"service-ca-75d99c484-6tqqj\" (UID: \"d6941c31-81b1-45b3-a568-363abcc7b706\") " pod="openshift-service-ca/service-ca-75d99c484-6tqqj"
Sep 09 12:49:11.018468 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.018418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d6941c31-81b1-45b3-a568-363abcc7b706-signing-key\") pod \"service-ca-75d99c484-6tqqj\" (UID: \"d6941c31-81b1-45b3-a568-363abcc7b706\") " pod="openshift-service-ca/service-ca-75d99c484-6tqqj"
Sep 09 12:49:11.018655 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.018475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzktp\" (UniqueName: \"kubernetes.io/projected/d6941c31-81b1-45b3-a568-363abcc7b706-kube-api-access-xzktp\") pod \"service-ca-75d99c484-6tqqj\" (UID: \"d6941c31-81b1-45b3-a568-363abcc7b706\") " pod="openshift-service-ca/service-ca-75d99c484-6tqqj"
Sep 09 12:49:11.018655 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.018508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d6941c31-81b1-45b3-a568-363abcc7b706-signing-cabundle\") pod \"service-ca-75d99c484-6tqqj\" (UID: \"d6941c31-81b1-45b3-a568-363abcc7b706\") " pod="openshift-service-ca/service-ca-75d99c484-6tqqj"
Sep 09 12:49:11.019300 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.019273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d6941c31-81b1-45b3-a568-363abcc7b706-signing-cabundle\") pod \"service-ca-75d99c484-6tqqj\" (UID: \"d6941c31-81b1-45b3-a568-363abcc7b706\") " pod="openshift-service-ca/service-ca-75d99c484-6tqqj"
Sep 09 12:49:11.022236 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.022215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d6941c31-81b1-45b3-a568-363abcc7b706-signing-key\") pod \"service-ca-75d99c484-6tqqj\" (UID: \"d6941c31-81b1-45b3-a568-363abcc7b706\") " pod="openshift-service-ca/service-ca-75d99c484-6tqqj"
Sep 09 12:49:11.028977 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.028954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzktp\" (UniqueName: \"kubernetes.io/projected/d6941c31-81b1-45b3-a568-363abcc7b706-kube-api-access-xzktp\") pod \"service-ca-75d99c484-6tqqj\" (UID: \"d6941c31-81b1-45b3-a568-363abcc7b706\") " pod="openshift-service-ca/service-ca-75d99c484-6tqqj"
Sep 09 12:49:11.104794 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.104687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-6tqqj"
Sep 09 12:49:11.219418 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.219371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:11.219418 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.219417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:11.219629 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.219491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:11.219629 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:11.219534    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:49:11.219629 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:11.219578    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:49:11.219629 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:11.219593    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls podName:739e03ea-ddf4-4d97-9552-851e87b324da nodeName:}" failed. No retries permitted until 2026-09-09 12:49:13.219579524 +0000 UTC m=+36.139470840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gtq25" (UID: "739e03ea-ddf4-4d97-9552-851e87b324da") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:49:11.219768 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:11.219636    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm podName:739e03ea-ddf4-4d97-9552-851e87b324da nodeName:}" failed. No retries permitted until 2026-09-09 12:49:13.219616379 +0000 UTC m=+36.139507725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gtq25" (UID: "739e03ea-ddf4-4d97-9552-851e87b324da") : configmap "kube-rbac-proxy" not found
Sep 09 12:49:11.219768 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:11.219672    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:49:11.219768 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:11.219718    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls podName:c3a0dfbf-070e-4c73-afd5-be6a620d8bb2 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:13.219706789 +0000 UTC m=+36.139598125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls") pod "dns-default-vhfsf" (UID: "c3a0dfbf-070e-4c73-afd5-be6a620d8bb2") : secret "dns-default-metrics-tls" not found
Sep 09 12:49:11.397643 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.397554    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-6tqqj"]
Sep 09 12:49:11.401331 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:11.401290    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6941c31_81b1_45b3_a568_363abcc7b706.slice/crio-ee87ae6ff480566f1d11546b4c148dea83b93d89777e152772e97df4a6a4e0eb WatchSource:0}: Error finding container ee87ae6ff480566f1d11546b4c148dea83b93d89777e152772e97df4a6a4e0eb: Status 404 returned error can't find the container with id ee87ae6ff480566f1d11546b4c148dea83b93d89777e152772e97df4a6a4e0eb
Sep 09 12:49:11.764877 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.764773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-6tqqj" event={"ID":"d6941c31-81b1-45b3-a568-363abcc7b706","Type":"ContainerStarted","Data":"ee87ae6ff480566f1d11546b4c148dea83b93d89777e152772e97df4a6a4e0eb"}
Sep 09 12:49:11.767829 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.767785    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f1a2660-8f7b-42e9-ae27-1f930df13a3c" containerID="1859e29309437c0bc51a0c08a180a9ad4874e4630357a919fb0b7b4fc8115831" exitCode=0
Sep 09 12:49:11.767947 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:11.767851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpb85" event={"ID":"8f1a2660-8f7b-42e9-ae27-1f930df13a3c","Type":"ContainerDied","Data":"1859e29309437c0bc51a0c08a180a9ad4874e4630357a919fb0b7b4fc8115831"}
Sep 09 12:49:12.773665 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:12.773627    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpb85" event={"ID":"8f1a2660-8f7b-42e9-ae27-1f930df13a3c","Type":"ContainerStarted","Data":"7d4b5cd0a0aace3e219642e12faeb696358c7704470775d24ed8ae54343e1619"}
Sep 09 12:49:12.795520 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:12.795466    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bpb85" podStartSLOduration=6.322000473 podStartE2EDuration="35.795449584s" totalImagesPullingTime="29.473449111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:12.794164415 +0000 UTC m=+35.714055753" watchObservedRunningTime="2026-09-09 12:49:12.795449584 +0000 UTC m=+35.715340920"
Sep 09 12:49:13.236184 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:13.236144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:13.236378 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:13.236205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:13.236378 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:13.236276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:13.236378 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:13.236308    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:49:13.236378 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:13.236368    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls podName:739e03ea-ddf4-4d97-9552-851e87b324da nodeName:}" failed. No retries permitted until 2026-09-09 12:49:17.236354503 +0000 UTC m=+40.156245835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gtq25" (UID: "739e03ea-ddf4-4d97-9552-851e87b324da") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:49:13.236378 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:13.236370    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:49:13.236586 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:13.236418    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls podName:c3a0dfbf-070e-4c73-afd5-be6a620d8bb2 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:17.236402439 +0000 UTC m=+40.156293759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls") pod "dns-default-vhfsf" (UID: "c3a0dfbf-070e-4c73-afd5-be6a620d8bb2") : secret "dns-default-metrics-tls" not found
Sep 09 12:49:13.236778 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:13.236757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/739e03ea-ddf4-4d97-9552-851e87b324da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:13.778595 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:13.778555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p8d9s" event={"ID":"7faebb79-50d6-4521-b829-ba8c91fb8a9b","Type":"ContainerStarted","Data":"1e183e168f8212ddc8fb8eabe440b8a43672b261dbb761e7c6db12fee8ac94ff"}
Sep 09 12:49:13.795686 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:13.795635    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p8d9s" podStartSLOduration=2.108878824 podStartE2EDuration="4.795618513s" totalImagesPullingTime="2.686739689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:13.794076597 +0000 UTC m=+36.713967949" watchObservedRunningTime="2026-09-09 12:49:13.795618513 +0000 UTC m=+36.715509851"
Sep 09 12:49:15.785393 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:15.785345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-6tqqj" event={"ID":"d6941c31-81b1-45b3-a568-363abcc7b706","Type":"ContainerStarted","Data":"8210d37dff499067395181a2074d3a354b5ea95116ff9a067ad38faecbcb054d"}
Sep 09 12:49:15.803434 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:15.803384    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-6tqqj" podStartSLOduration=2.443944298 podStartE2EDuration="5.803369672s" totalImagesPullingTime="3.359425374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:15.802398574 +0000 UTC m=+38.722289913" watchObservedRunningTime="2026-09-09 12:49:15.803369672 +0000 UTC m=+38.723261002"
Sep 09 12:49:17.273957 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:17.273921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:17.274353 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:17.273983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:17.274353 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:17.274071    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:49:17.274353 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:17.274073    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:49:17.274353 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:17.274145    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls podName:c3a0dfbf-070e-4c73-afd5-be6a620d8bb2 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:25.274126253 +0000 UTC m=+48.194017591 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls") pod "dns-default-vhfsf" (UID: "c3a0dfbf-070e-4c73-afd5-be6a620d8bb2") : secret "dns-default-metrics-tls" not found
Sep 09 12:49:17.274353 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:17.274164    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls podName:739e03ea-ddf4-4d97-9552-851e87b324da nodeName:}" failed. No retries permitted until 2026-09-09 12:49:25.274155879 +0000 UTC m=+48.194047212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gtq25" (UID: "739e03ea-ddf4-4d97-9552-851e87b324da") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:49:25.336571 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:25.336534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:25.336977 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:25.336590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:25.339951 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:25.339923    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/739e03ea-ddf4-4d97-9552-851e87b324da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gtq25\" (UID: \"739e03ea-ddf4-4d97-9552-851e87b324da\") " pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:25.340070 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:25.339930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3a0dfbf-070e-4c73-afd5-be6a620d8bb2-metrics-tls\") pod \"dns-default-vhfsf\" (UID: \"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2\") " pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:25.363072 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:25.363050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:25.395229 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:25.395206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gtq25"
Sep 09 12:49:25.676037 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:25.675971    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vhfsf"]
Sep 09 12:49:25.678844 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:25.678803    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a0dfbf_070e_4c73_afd5_be6a620d8bb2.slice/crio-b8a6c131b9166758a30f0d27083a0c72ae6478374e2cba0f9f08496264dcaa8c WatchSource:0}: Error finding container b8a6c131b9166758a30f0d27083a0c72ae6478374e2cba0f9f08496264dcaa8c: Status 404 returned error can't find the container with id b8a6c131b9166758a30f0d27083a0c72ae6478374e2cba0f9f08496264dcaa8c
Sep 09 12:49:25.698622 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:25.698598    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gtq25"]
Sep 09 12:49:25.702733 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:25.702709    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739e03ea_ddf4_4d97_9552_851e87b324da.slice/crio-eedfc6d9f793fdee62878178dcda768a81884c2563f854453c5e2abf9e0db0c0 WatchSource:0}: Error finding container eedfc6d9f793fdee62878178dcda768a81884c2563f854453c5e2abf9e0db0c0: Status 404 returned error can't find the container with id eedfc6d9f793fdee62878178dcda768a81884c2563f854453c5e2abf9e0db0c0
Sep 09 12:49:25.810942 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:25.810914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gtq25" event={"ID":"739e03ea-ddf4-4d97-9552-851e87b324da","Type":"ContainerStarted","Data":"9ca49ea47c104d18c35791bfc4a8162c8df4c63dfd4c12410de2429f390b9e65"}
Sep 09 12:49:25.811033 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:25.810982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gtq25" event={"ID":"739e03ea-ddf4-4d97-9552-851e87b324da","Type":"ContainerStarted","Data":"eedfc6d9f793fdee62878178dcda768a81884c2563f854453c5e2abf9e0db0c0"}
Sep 09 12:49:25.812033 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:25.812014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhfsf" event={"ID":"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2","Type":"ContainerStarted","Data":"b8a6c131b9166758a30f0d27083a0c72ae6478374e2cba0f9f08496264dcaa8c"}
Sep 09 12:49:26.816432 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:26.816399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gtq25" event={"ID":"739e03ea-ddf4-4d97-9552-851e87b324da","Type":"ContainerStarted","Data":"ac33d513b408fd706f8a5dc1734b432161002db6956017968d2e0d8e66e161b2"}
Sep 09 12:49:28.824511 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:28.824479    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhfsf" event={"ID":"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2","Type":"ContainerStarted","Data":"0e99e2d059c4a5a24a12125159e497d90a47492f4d3813508bc8888bb3b29d2c"}
Sep 09 12:49:28.824934 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:28.824542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhfsf" event={"ID":"c3a0dfbf-070e-4c73-afd5-be6a620d8bb2","Type":"ContainerStarted","Data":"323c400e10c1627184f439f4952a1bd021e34c68283fbb0a5a60f01f21648b27"}
Sep 09 12:49:28.824934 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:28.824571    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:28.826722 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:28.826702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gtq25" event={"ID":"739e03ea-ddf4-4d97-9552-851e87b324da","Type":"ContainerStarted","Data":"4d19b6541f3e3563721e73a0a3e5769b197b5c64bd0c7cfeedc5cff842b2863a"}
Sep 09 12:49:28.840737 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:28.840694    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vhfsf" podStartSLOduration=16.873434921 podStartE2EDuration="19.840680396s" totalImagesPullingTime="2.967245475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:28.839626026 +0000 UTC m=+51.759517376" watchObservedRunningTime="2026-09-09 12:49:28.840680396 +0000 UTC m=+51.760571736"
Sep 09 12:49:28.855762 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:28.855716    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gtq25" podStartSLOduration=17.116079356 podStartE2EDuration="19.855699626s" totalImagesPullingTime="2.73962027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:49:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:28.855652588 +0000 UTC m=+51.775543926" watchObservedRunningTime="2026-09-09 12:49:28.855699626 +0000 UTC m=+51.775590965"
Sep 09 12:49:32.718187 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.718152    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2"]
Sep 09 12:49:32.720073 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.720057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2"
Sep 09 12:49:32.724745 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.724715    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 12:49:32.724745 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.724727    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-h28zz\""
Sep 09 12:49:32.724745 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.724718    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 12:49:32.725008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.724718    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 12:49:32.725008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.724790    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 12:49:32.734579 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.734554    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2"]
Sep 09 12:49:32.792629 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.792603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrj9z\" (UniqueName: \"kubernetes.io/projected/b8890975-4cce-4d3f-aa3b-97082856c7c6-kube-api-access-jrj9z\") pod \"managed-serviceaccount-addon-agent-5669897654-7zfb2\" (UID: \"b8890975-4cce-4d3f-aa3b-97082856c7c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2"
Sep 09 12:49:32.792793 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.792694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8890975-4cce-4d3f-aa3b-97082856c7c6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5669897654-7zfb2\" (UID: \"b8890975-4cce-4d3f-aa3b-97082856c7c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2"
Sep 09 12:49:32.860935 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.860907    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b45d9f7f5-bzzcr"]
Sep 09 12:49:32.862753 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.862738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.866661 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.866642    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 12:49:32.866774 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.866671    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 12:49:32.866903 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.866885    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qkhdl\""
Sep 09 12:49:32.867370 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.867349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 12:49:32.881517 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.881499    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 12:49:32.893058 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.893035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-trusted-ca\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.893140 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.893068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27gxr\" (UniqueName: \"kubernetes.io/projected/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-kube-api-access-27gxr\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.893140 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.893089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-image-registry-private-configuration\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.893140 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.893123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrj9z\" (UniqueName: \"kubernetes.io/projected/b8890975-4cce-4d3f-aa3b-97082856c7c6-kube-api-access-jrj9z\") pod \"managed-serviceaccount-addon-agent-5669897654-7zfb2\" (UID: \"b8890975-4cce-4d3f-aa3b-97082856c7c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2"
Sep 09 12:49:32.893238 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.893145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-installation-pull-secrets\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.893238 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.893211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-ca-trust-extracted\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.893238 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.893226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-registry-certificates\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.893442 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.893242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-registry-tls\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.893442 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.893261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-bound-sa-token\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.893442 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.893302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8890975-4cce-4d3f-aa3b-97082856c7c6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5669897654-7zfb2\" (UID: \"b8890975-4cce-4d3f-aa3b-97082856c7c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2"
Sep 09 12:49:32.895766 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.895741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8890975-4cce-4d3f-aa3b-97082856c7c6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5669897654-7zfb2\" (UID: \"b8890975-4cce-4d3f-aa3b-97082856c7c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2"
Sep 09 12:49:32.903638 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.903617    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b45d9f7f5-bzzcr"]
Sep 09 12:49:32.915494 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.915470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrj9z\" (UniqueName: \"kubernetes.io/projected/b8890975-4cce-4d3f-aa3b-97082856c7c6-kube-api-access-jrj9z\") pod \"managed-serviceaccount-addon-agent-5669897654-7zfb2\" (UID: \"b8890975-4cce-4d3f-aa3b-97082856c7c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2"
Sep 09 12:49:32.993774 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.993701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-registry-tls\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.993774 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.993732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-bound-sa-token\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.993774 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.993749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-trusted-ca\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.993774 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.993767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27gxr\" (UniqueName: \"kubernetes.io/projected/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-kube-api-access-27gxr\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.994083 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.993786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-image-registry-private-configuration\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.994083 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.993803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-installation-pull-secrets\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.994083 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.993924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-ca-trust-extracted\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.994083 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.993952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-registry-certificates\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.994479 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.994444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-ca-trust-extracted\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.995121 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.995103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-trusted-ca\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.995327 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.995307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-registry-certificates\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.996632 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.996609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-registry-tls\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.996905 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.996886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-installation-pull-secrets\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:32.997004 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:32.996985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-image-registry-private-configuration\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:33.002233 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:33.002191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-bound-sa-token\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:33.002586 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:33.002568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27gxr\" (UniqueName: \"kubernetes.io/projected/95b8a79e-17b1-4b49-9cb9-bfb90cef9672-kube-api-access-27gxr\") pod \"image-registry-b45d9f7f5-bzzcr\" (UID: \"95b8a79e-17b1-4b49-9cb9-bfb90cef9672\") " pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:33.042446 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:33.042387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2"
Sep 09 12:49:33.173055 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:33.173026    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:33.308050 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:33.308022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2"]
Sep 09 12:49:33.310727 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:33.310694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8890975_4cce_4d3f_aa3b_97082856c7c6.slice/crio-842d00427198160c1194c6ea93c845f6b080740104b50c5856409a3069f221ad WatchSource:0}: Error finding container 842d00427198160c1194c6ea93c845f6b080740104b50c5856409a3069f221ad: Status 404 returned error can't find the container with id 842d00427198160c1194c6ea93c845f6b080740104b50c5856409a3069f221ad
Sep 09 12:49:33.442951 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:33.442919    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b45d9f7f5-bzzcr"]
Sep 09 12:49:33.446741 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:33.446707    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b8a79e_17b1_4b49_9cb9_bfb90cef9672.slice/crio-147c8fd30629af3d4d0b224340d237d0223a00e92e1225c744fcec7f35d7fef9 WatchSource:0}: Error finding container 147c8fd30629af3d4d0b224340d237d0223a00e92e1225c744fcec7f35d7fef9: Status 404 returned error can't find the container with id 147c8fd30629af3d4d0b224340d237d0223a00e92e1225c744fcec7f35d7fef9
Sep 09 12:49:33.845610 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:33.845572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr" event={"ID":"95b8a79e-17b1-4b49-9cb9-bfb90cef9672","Type":"ContainerStarted","Data":"5a2a17d9ea1c9b7be8229b6e5cac70f57d9e5ebfcd753b453e1c9818f7793d4e"}
Sep 09 12:49:33.846061 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:33.845633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr" event={"ID":"95b8a79e-17b1-4b49-9cb9-bfb90cef9672","Type":"ContainerStarted","Data":"147c8fd30629af3d4d0b224340d237d0223a00e92e1225c744fcec7f35d7fef9"}
Sep 09 12:49:33.846061 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:33.845661    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:33.846993 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:33.846968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2" event={"ID":"b8890975-4cce-4d3f-aa3b-97082856c7c6","Type":"ContainerStarted","Data":"842d00427198160c1194c6ea93c845f6b080740104b50c5856409a3069f221ad"}
Sep 09 12:49:33.868798 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:33.868735    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr" podStartSLOduration=1.868714795 podStartE2EDuration="1.868714795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:33.868192324 +0000 UTC m=+56.788083663" watchObservedRunningTime="2026-09-09 12:49:33.868714795 +0000 UTC m=+56.788606135"
Sep 09 12:49:34.920778 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:34.920718    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2"]
Sep 09 12:49:34.942147 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:34.942114    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2"]
Sep 09 12:49:34.942322 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:34.942235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2"
Sep 09 12:49:34.945010 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:34.944983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 12:49:34.945213 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:34.945182    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-brlwq\""
Sep 09 12:49:35.006341 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:35.006300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4d1687c-864d-45fa-b24f-ce8af11ed168-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-7b7k2\" (UID: \"f4d1687c-864d-45fa-b24f-ce8af11ed168\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2"
Sep 09 12:49:35.107791 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:35.107755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4d1687c-864d-45fa-b24f-ce8af11ed168-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-7b7k2\" (UID: \"f4d1687c-864d-45fa-b24f-ce8af11ed168\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2"
Sep 09 12:49:35.110954 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:35.110925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4d1687c-864d-45fa-b24f-ce8af11ed168-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-7b7k2\" (UID: \"f4d1687c-864d-45fa-b24f-ce8af11ed168\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2"
Sep 09 12:49:35.255049 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:35.254960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2"
Sep 09 12:49:36.198616 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:36.198532    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2"]
Sep 09 12:49:36.202043 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:36.202005    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d1687c_864d_45fa_b24f_ce8af11ed168.slice/crio-9806e0ce5ef4d6e9498222a41d01fd5859bf4d47b75b1049f4ae4b1f136dcacd WatchSource:0}: Error finding container 9806e0ce5ef4d6e9498222a41d01fd5859bf4d47b75b1049f4ae4b1f136dcacd: Status 404 returned error can't find the container with id 9806e0ce5ef4d6e9498222a41d01fd5859bf4d47b75b1049f4ae4b1f136dcacd
Sep 09 12:49:36.757193 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:36.757117    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cq52"
Sep 09 12:49:36.861360 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:36.861306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2" event={"ID":"b8890975-4cce-4d3f-aa3b-97082856c7c6","Type":"ContainerStarted","Data":"8d291cb424f2e73120ca106ef3bbc5eb6606fb75750520af1f559591309f96e8"}
Sep 09 12:49:36.862715 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:36.862685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2" event={"ID":"f4d1687c-864d-45fa-b24f-ce8af11ed168","Type":"ContainerStarted","Data":"9806e0ce5ef4d6e9498222a41d01fd5859bf4d47b75b1049f4ae4b1f136dcacd"}
Sep 09 12:49:36.876882 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:36.876809    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5669897654-7zfb2" podStartSLOduration=2.217252343 podStartE2EDuration="4.876791973s" totalImagesPullingTime="2.65953963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:36.876185692 +0000 UTC m=+59.796077034" watchObservedRunningTime="2026-09-09 12:49:36.876791973 +0000 UTC m=+59.796683302"
Sep 09 12:49:37.867876 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:37.867830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2" event={"ID":"f4d1687c-864d-45fa-b24f-ce8af11ed168","Type":"ContainerStarted","Data":"5ed240c8c90734ebcaa9a6da1f6544e5f8f9b35d9694771ad6ccb7fcd40224e4"}
Sep 09 12:49:37.882961 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:37.882897    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2" podStartSLOduration=2.64740549 podStartE2EDuration="3.882874201s" totalImagesPullingTime="1.235468711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:37.882787811 +0000 UTC m=+60.802679152" watchObservedRunningTime="2026-09-09 12:49:37.882874201 +0000 UTC m=+60.802765540"
Sep 09 12:49:38.832746 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:38.832715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vhfsf"
Sep 09 12:49:38.871736 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:38.871709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2"
Sep 09 12:49:38.876554 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:38.876529    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-7b7k2"
Sep 09 12:49:39.974197 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:39.974158    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"]
Sep 09 12:49:39.977094 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:39.977070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:39.981476 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:39.980790    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vb2d5\""
Sep 09 12:49:39.981716 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:39.981691    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 12:49:39.982256 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:39.982232    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 12:49:39.982408 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:39.982390    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 12:49:39.982538 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:39.982519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 12:49:39.982662 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:39.982524    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 12:49:39.984971 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:39.984950    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"]
Sep 09 12:49:40.043510 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.043465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fr4n\" (UniqueName: \"kubernetes.io/projected/8086d1db-13a0-41e5-8775-2a1bd44137c2-kube-api-access-6fr4n\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.043692 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.043544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8086d1db-13a0-41e5-8775-2a1bd44137c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.043692 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.043568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8086d1db-13a0-41e5-8775-2a1bd44137c2-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.043692 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.043600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8086d1db-13a0-41e5-8775-2a1bd44137c2-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.144900 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.144857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fr4n\" (UniqueName: \"kubernetes.io/projected/8086d1db-13a0-41e5-8775-2a1bd44137c2-kube-api-access-6fr4n\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.145011 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.144961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8086d1db-13a0-41e5-8775-2a1bd44137c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.145011 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.144993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8086d1db-13a0-41e5-8775-2a1bd44137c2-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.145071 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.145047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8086d1db-13a0-41e5-8775-2a1bd44137c2-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.145189 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:40.145176    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 09 12:49:40.145285 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:40.145245    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8086d1db-13a0-41e5-8775-2a1bd44137c2-prometheus-operator-tls podName:8086d1db-13a0-41e5-8775-2a1bd44137c2 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:40.645226364 +0000 UTC m=+63.565117695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8086d1db-13a0-41e5-8775-2a1bd44137c2-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-9r5z6" (UID: "8086d1db-13a0-41e5-8775-2a1bd44137c2") : secret "prometheus-operator-tls" not found
Sep 09 12:49:40.145711 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.145686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8086d1db-13a0-41e5-8775-2a1bd44137c2-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.147580 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.147554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8086d1db-13a0-41e5-8775-2a1bd44137c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.153807 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.153777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fr4n\" (UniqueName: \"kubernetes.io/projected/8086d1db-13a0-41e5-8775-2a1bd44137c2-kube-api-access-6fr4n\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.649485 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.649445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8086d1db-13a0-41e5-8775-2a1bd44137c2-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.652015 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.651983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8086d1db-13a0-41e5-8775-2a1bd44137c2-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-9r5z6\" (UID: \"8086d1db-13a0-41e5-8775-2a1bd44137c2\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:40.889883 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:40.889849    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"
Sep 09 12:49:41.191146 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:41.191064    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-9r5z6"]
Sep 09 12:49:41.194297 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:41.194270    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8086d1db_13a0_41e5_8775_2a1bd44137c2.slice/crio-3bc8ff984d64a68cd005fa9a86baad188cefdb6dbb5b7b05989b1cd7e2db9727 WatchSource:0}: Error finding container 3bc8ff984d64a68cd005fa9a86baad188cefdb6dbb5b7b05989b1cd7e2db9727: Status 404 returned error can't find the container with id 3bc8ff984d64a68cd005fa9a86baad188cefdb6dbb5b7b05989b1cd7e2db9727
Sep 09 12:49:41.883132 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:41.883090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6" event={"ID":"8086d1db-13a0-41e5-8775-2a1bd44137c2","Type":"ContainerStarted","Data":"3bc8ff984d64a68cd005fa9a86baad188cefdb6dbb5b7b05989b1cd7e2db9727"}
Sep 09 12:49:42.364577 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.364535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:49:42.364577 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.364574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:49:42.367516 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.367476    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 12:49:42.368262 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.368243    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 12:49:42.377761 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.377733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2014baf8-52ee-420c-89d0-205989b32b87-original-pull-secret\") pod \"global-pull-secret-syncer-7jcs4\" (UID: \"2014baf8-52ee-420c-89d0-205989b32b87\") " pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:49:42.377911 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.377850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9-metrics-certs\") pod \"network-metrics-daemon-gjfxj\" (UID: \"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9\") " pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:49:42.396027 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.395996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7jcs4"
Sep 09 12:49:42.409312 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.409280    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ns67g\""
Sep 09 12:49:42.416745 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.416716    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjfxj"
Sep 09 12:49:42.465475 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.465434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njclh\" (UniqueName: \"kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh\") pod \"network-check-target-2n2wj\" (UID: \"ab6539a3-0367-425b-9dc0-cc894772df3a\") " pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:42.468460 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.468438    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 12:49:42.479393 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.479354    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 12:49:42.490197 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.490167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njclh\" (UniqueName: \"kubernetes.io/projected/ab6539a3-0367-425b-9dc0-cc894772df3a-kube-api-access-njclh\") pod \"network-check-target-2n2wj\" (UID: \"ab6539a3-0367-425b-9dc0-cc894772df3a\") " pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:42.704696 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.704663    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qcx5j\""
Sep 09 12:49:42.712769 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.712739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:42.850249 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.850217    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7jcs4"]
Sep 09 12:49:42.854510 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:42.854479    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gjfxj"]
Sep 09 12:49:42.919155 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:42.919068    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2014baf8_52ee_420c_89d0_205989b32b87.slice/crio-d1a47336845d62364f71b64a698dbe7258eb7d001cb20e6974d19868ae126119 WatchSource:0}: Error finding container d1a47336845d62364f71b64a698dbe7258eb7d001cb20e6974d19868ae126119: Status 404 returned error can't find the container with id d1a47336845d62364f71b64a698dbe7258eb7d001cb20e6974d19868ae126119
Sep 09 12:49:42.919673 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:42.919651    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8d8768_b0e0_44f5_affc_4ae8a8bbc3a9.slice/crio-c992b474754dbd3ef9326fc4e1edbcee2041690396ca6e5be4036092f8c834d3 WatchSource:0}: Error finding container c992b474754dbd3ef9326fc4e1edbcee2041690396ca6e5be4036092f8c834d3: Status 404 returned error can't find the container with id c992b474754dbd3ef9326fc4e1edbcee2041690396ca6e5be4036092f8c834d3
Sep 09 12:49:43.011956 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.011920    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2n2wj"]
Sep 09 12:49:43.018115 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:43.017797    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6539a3_0367_425b_9dc0_cc894772df3a.slice/crio-ba6eb87db9666ced68bc72f2a2d22c4e048717a639c94076f004bcee23f8ddb9 WatchSource:0}: Error finding container ba6eb87db9666ced68bc72f2a2d22c4e048717a639c94076f004bcee23f8ddb9: Status 404 returned error can't find the container with id ba6eb87db9666ced68bc72f2a2d22c4e048717a639c94076f004bcee23f8ddb9
Sep 09 12:49:43.019209 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.019185    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-mjfdh"]
Sep 09 12:49:43.048658 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.048632    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-mjfdh"]
Sep 09 12:49:43.048991 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.048732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-mjfdh"
Sep 09 12:49:43.051637 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.051608    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 12:49:43.051742 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.051649    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-f87td\""
Sep 09 12:49:43.051917 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.051903    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 12:49:43.069753 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.069722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdjqm\" (UniqueName: \"kubernetes.io/projected/f09261ce-ec57-49c9-b05c-cdf587fc4853-kube-api-access-mdjqm\") pod \"downloads-85d98c998f-mjfdh\" (UID: \"f09261ce-ec57-49c9-b05c-cdf587fc4853\") " pod="openshift-console/downloads-85d98c998f-mjfdh"
Sep 09 12:49:43.171089 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.170994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdjqm\" (UniqueName: \"kubernetes.io/projected/f09261ce-ec57-49c9-b05c-cdf587fc4853-kube-api-access-mdjqm\") pod \"downloads-85d98c998f-mjfdh\" (UID: \"f09261ce-ec57-49c9-b05c-cdf587fc4853\") " pod="openshift-console/downloads-85d98c998f-mjfdh"
Sep 09 12:49:43.180346 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.180316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdjqm\" (UniqueName: \"kubernetes.io/projected/f09261ce-ec57-49c9-b05c-cdf587fc4853-kube-api-access-mdjqm\") pod \"downloads-85d98c998f-mjfdh\" (UID: \"f09261ce-ec57-49c9-b05c-cdf587fc4853\") " pod="openshift-console/downloads-85d98c998f-mjfdh"
Sep 09 12:49:43.358482 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.358450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-mjfdh"
Sep 09 12:49:43.654231 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.654193    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-mjfdh"]
Sep 09 12:49:43.656433 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:43.656400    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09261ce_ec57_49c9_b05c_cdf587fc4853.slice/crio-63658d5a3953e06747b1ae280274e48be44a0ce76e4ed401829513b38fc4d686 WatchSource:0}: Error finding container 63658d5a3953e06747b1ae280274e48be44a0ce76e4ed401829513b38fc4d686: Status 404 returned error can't find the container with id 63658d5a3953e06747b1ae280274e48be44a0ce76e4ed401829513b38fc4d686
Sep 09 12:49:43.892802 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.892756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2n2wj" event={"ID":"ab6539a3-0367-425b-9dc0-cc894772df3a","Type":"ContainerStarted","Data":"ba6eb87db9666ced68bc72f2a2d22c4e048717a639c94076f004bcee23f8ddb9"}
Sep 09 12:49:43.894549 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.894513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gjfxj" event={"ID":"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9","Type":"ContainerStarted","Data":"c992b474754dbd3ef9326fc4e1edbcee2041690396ca6e5be4036092f8c834d3"}
Sep 09 12:49:43.896692 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.896645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7jcs4" event={"ID":"2014baf8-52ee-420c-89d0-205989b32b87","Type":"ContainerStarted","Data":"d1a47336845d62364f71b64a698dbe7258eb7d001cb20e6974d19868ae126119"}
Sep 09 12:49:43.899696 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.899643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6" event={"ID":"8086d1db-13a0-41e5-8775-2a1bd44137c2","Type":"ContainerStarted","Data":"19729fb06b6203901672e896f07ced71b5d1ecfaf7374931c6b1ab8f49ba17f3"}
Sep 09 12:49:43.899960 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.899732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6" event={"ID":"8086d1db-13a0-41e5-8775-2a1bd44137c2","Type":"ContainerStarted","Data":"31ff7a860a2ac0c30a84456d1d29637082ca8d92bbf423611bc93fa36e7164c8"}
Sep 09 12:49:43.903294 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.903257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-mjfdh" event={"ID":"f09261ce-ec57-49c9-b05c-cdf587fc4853","Type":"ContainerStarted","Data":"63658d5a3953e06747b1ae280274e48be44a0ce76e4ed401829513b38fc4d686"}
Sep 09 12:49:43.916533 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:43.916481    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-9r5z6" podStartSLOduration=3.161629121 podStartE2EDuration="4.916459168s" totalImagesPullingTime="1.754830047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:43.916026894 +0000 UTC m=+66.835918234" watchObservedRunningTime="2026-09-09 12:49:43.916459168 +0000 UTC m=+66.836350507"
Sep 09 12:49:44.912417 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:44.912378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gjfxj" event={"ID":"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9","Type":"ContainerStarted","Data":"dbcc4aeea035dd330d9238bb8569b335226d807bc91bb4dea9417e117c9243f3"}
Sep 09 12:49:44.912870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:44.912436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gjfxj" event={"ID":"fd8d8768-b0e0-44f5-affc-4ae8a8bbc3a9","Type":"ContainerStarted","Data":"ada68832369e9e6d7ce89483a83b790b2dde8694d626ed0f3b12457cbc313832"}
Sep 09 12:49:44.928652 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:44.928589    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gjfxj" podStartSLOduration=66.58134785 podStartE2EDuration="1m7.928567527s" totalImagesPullingTime="1.347219677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:44.926907396 +0000 UTC m=+67.846798729" watchObservedRunningTime="2026-09-09 12:49:44.928567527 +0000 UTC m=+67.848458870"
Sep 09 12:49:45.343837 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.343215    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"]
Sep 09 12:49:45.356834 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.346048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.356834 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.349537    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 12:49:45.356834 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.349764    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cwg89\""
Sep 09 12:49:45.356834 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.349862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 12:49:45.369836 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.367585    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"]
Sep 09 12:49:45.375036 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.373524    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qb94p"]
Sep 09 12:49:45.377489 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.376155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.381719 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.380942    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t4lg7\""
Sep 09 12:49:45.381719 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.381197    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 12:49:45.381719 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.381454    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 12:49:45.381719 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.381652    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 12:49:45.395254 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.395022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.395254 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.395083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.395254 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.395111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.395254 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.395136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrxx\" (UniqueName: \"kubernetes.io/projected/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-kube-api-access-vbrxx\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.498535 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.498726 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.498726 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-accelerators-collector-config\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.498726 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.498726 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.498726 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-tls\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.498726 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53f9d380-aeec-4335-995d-5cd1df5e0026-metrics-client-ca\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.498726 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjlx6\" (UniqueName: \"kubernetes.io/projected/53f9d380-aeec-4335-995d-5cd1df5e0026-kube-api-access-zjlx6\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.499274 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrxx\" (UniqueName: \"kubernetes.io/projected/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-kube-api-access-vbrxx\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.499274 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498784    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-textfile\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.499274 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-wtmp\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.499274 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53f9d380-aeec-4335-995d-5cd1df5e0026-sys\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.499274 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.498895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53f9d380-aeec-4335-995d-5cd1df5e0026-root\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.501268 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.501238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.517142 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.502226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.517142 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.504262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.527080 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.527020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrxx\" (UniqueName: \"kubernetes.io/projected/8345b31d-f1bd-42ee-bf5e-b24dc175ce63-kube-api-access-vbrxx\") pod \"openshift-state-metrics-657fd6c54-l5zws\" (UID: \"8345b31d-f1bd-42ee-bf5e-b24dc175ce63\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.600178 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53f9d380-aeec-4335-995d-5cd1df5e0026-sys\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.600178 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53f9d380-aeec-4335-995d-5cd1df5e0026-root\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.600178 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.600457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-accelerators-collector-config\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.600457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-tls\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.600457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53f9d380-aeec-4335-995d-5cd1df5e0026-metrics-client-ca\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.600457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjlx6\" (UniqueName: \"kubernetes.io/projected/53f9d380-aeec-4335-995d-5cd1df5e0026-kube-api-access-zjlx6\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.600457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-textfile\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.600457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-wtmp\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.600457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-wtmp\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.600802 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53f9d380-aeec-4335-995d-5cd1df5e0026-sys\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.600802 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.600488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53f9d380-aeec-4335-995d-5cd1df5e0026-root\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.601533 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.601493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-textfile\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.601640 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:45.601619    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 12:49:45.602039 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:45.602020    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-tls podName:53f9d380-aeec-4335-995d-5cd1df5e0026 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:46.101997857 +0000 UTC m=+69.021889187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-tls") pod "node-exporter-qb94p" (UID: "53f9d380-aeec-4335-995d-5cd1df5e0026") : secret "node-exporter-tls" not found
Sep 09 12:49:45.602184 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.602161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53f9d380-aeec-4335-995d-5cd1df5e0026-metrics-client-ca\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.602686 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.602652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-accelerators-collector-config\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.609214 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.609141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.611310 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.611285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjlx6\" (UniqueName: \"kubernetes.io/projected/53f9d380-aeec-4335-995d-5cd1df5e0026-kube-api-access-zjlx6\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:45.661208 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.660913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"
Sep 09 12:49:45.976167 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:45.975346    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws"]
Sep 09 12:49:46.105840 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.105777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-tls\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:46.109553 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.109521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53f9d380-aeec-4335-995d-5cd1df5e0026-node-exporter-tls\") pod \"node-exporter-qb94p\" (UID: \"53f9d380-aeec-4335-995d-5cd1df5e0026\") " pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:46.293063 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.292975    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qb94p"
Sep 09 12:49:46.429889 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.429842    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:49:46.433325 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.433116    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.438728 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.438692    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 12:49:46.439014 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.438991    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 12:49:46.439272 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.439257    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 12:49:46.439316 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.439304    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 12:49:46.442223 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.439530    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 12:49:46.442223 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.439757    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-26v94\""
Sep 09 12:49:46.442223 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.442045    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 12:49:46.444709 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.444687    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 12:49:46.444910 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.444886    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 12:49:46.445087 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.444996    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 12:49:46.453717 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.453609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:49:46.509130 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509308 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-config-volume\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509308 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-web-config\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509308 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509308 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-config-out\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509308 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509308 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509308 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509625 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgv5j\" (UniqueName: \"kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-kube-api-access-kgv5j\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509625 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509625 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509625 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.509625 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.509438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgv5j\" (UniqueName: \"kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-kube-api-access-kgv5j\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-config-volume\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-web-config\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-config-out\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.610846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:46.611003    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 12:49:46.611870 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:46.611073    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls podName:29695e19-8030-4d99-82c1-56bdba6d3689 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:47.111050386 +0000 UTC m=+70.030941706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689") : secret "alertmanager-main-tls" not found
Sep 09 12:49:46.612738 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.612596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.613877 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.613848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.614008 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.613949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.616860 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.616790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.620060 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.620033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-config-volume\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.622758 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.622330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-config-out\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.622758 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.622588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.622758 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.622588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.622758 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.622686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.623041 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.623014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-web-config\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.623041 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.623019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:46.625157 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:46.625102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgv5j\" (UniqueName: \"kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-kube-api-access-kgv5j\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:47.115451 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.115408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:47.115862 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:47.115573    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 12:49:47.115862 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:49:47.115647    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls podName:29695e19-8030-4d99-82c1-56bdba6d3689 nodeName:}" failed. No retries permitted until 2026-09-09 12:49:48.115628742 +0000 UTC m=+71.035520059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689") : secret "alertmanager-main-tls" not found
Sep 09 12:49:47.330674 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.330637    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"]
Sep 09 12:49:47.335419 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.335388    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.338289 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.338264    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 12:49:47.338421 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.338293    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4p74unuauq1u3\""
Sep 09 12:49:47.338610 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.338591    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 12:49:47.338610 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.338604    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-p8jn9\""
Sep 09 12:49:47.338733 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.338617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 12:49:47.338733 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.338679    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 12:49:47.338733 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.338725    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 12:49:47.347447 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.347353    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"]
Sep 09 12:49:47.417760 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.417668    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-grpc-tls\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.417760 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.417747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.417994 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.417778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a65c5be9-4322-4a18-be47-6a49e215ad19-metrics-client-ca\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.417994 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.417808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.417994 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.417863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.417994 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.417901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-tls\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.417994 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.417938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.417994 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.417966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zmj\" (UniqueName: \"kubernetes.io/projected/a65c5be9-4322-4a18-be47-6a49e215ad19-kube-api-access-f2zmj\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.519749 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.519128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-grpc-tls\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.519749 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.519197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.519749 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.519328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a65c5be9-4322-4a18-be47-6a49e215ad19-metrics-client-ca\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.519749 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.519384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.519749 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.519427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.519749 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.519468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-tls\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.519749 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.519525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.519749 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.519558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zmj\" (UniqueName: \"kubernetes.io/projected/a65c5be9-4322-4a18-be47-6a49e215ad19-kube-api-access-f2zmj\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.520978 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.520949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a65c5be9-4322-4a18-be47-6a49e215ad19-metrics-client-ca\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.522916 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.522892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.523092 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.523070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-tls\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.523619 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.523597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.524116 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.524093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-grpc-tls\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.524523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.524503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.525216 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.525188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a65c5be9-4322-4a18-be47-6a49e215ad19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.527964 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.527941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zmj\" (UniqueName: \"kubernetes.io/projected/a65c5be9-4322-4a18-be47-6a49e215ad19-kube-api-access-f2zmj\") pod \"thanos-querier-7f4b64f9fb-gmdz6\" (UID: \"a65c5be9-4322-4a18-be47-6a49e215ad19\") " pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:47.648239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:47.648205    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:48.126587 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:48.126507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:48.130792 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:48.130753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:48.261872 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:48.261811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:49:48.916980 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:48.916932    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8345b31d_f1bd_42ee_bf5e_b24dc175ce63.slice/crio-412e9120db36f5225efda132d5d880de43647e7054228565cd8027c2ac939208 WatchSource:0}: Error finding container 412e9120db36f5225efda132d5d880de43647e7054228565cd8027c2ac939208: Status 404 returned error can't find the container with id 412e9120db36f5225efda132d5d880de43647e7054228565cd8027c2ac939208
Sep 09 12:49:48.936425 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:48.936381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws" event={"ID":"8345b31d-f1bd-42ee-bf5e-b24dc175ce63","Type":"ContainerStarted","Data":"412e9120db36f5225efda132d5d880de43647e7054228565cd8027c2ac939208"}
Sep 09 12:49:48.980862 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:48.980553    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f9d380_aeec_4335_995d_5cd1df5e0026.slice/crio-baa3ac7556348ee4ef10c0ed91c9a1989ebc9600054845170c32e9e3cc19241a WatchSource:0}: Error finding container baa3ac7556348ee4ef10c0ed91c9a1989ebc9600054845170c32e9e3cc19241a: Status 404 returned error can't find the container with id baa3ac7556348ee4ef10c0ed91c9a1989ebc9600054845170c32e9e3cc19241a
Sep 09 12:49:49.259720 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.259667    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"]
Sep 09 12:49:49.264284 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:49.264252    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65c5be9_4322_4a18_be47_6a49e215ad19.slice/crio-e8fad907f3139d4a861d52b27106ed9d41767f4344dfd0c1fd54395833de6923 WatchSource:0}: Error finding container e8fad907f3139d4a861d52b27106ed9d41767f4344dfd0c1fd54395833de6923: Status 404 returned error can't find the container with id e8fad907f3139d4a861d52b27106ed9d41767f4344dfd0c1fd54395833de6923
Sep 09 12:49:49.286104 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.285942    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:49:49.287841 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:49.287784    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29695e19_8030_4d99_82c1_56bdba6d3689.slice/crio-a71babdb3341abd73a85d16b7a26da94d1b6565b9e69c27b288b9209dcea9980 WatchSource:0}: Error finding container a71babdb3341abd73a85d16b7a26da94d1b6565b9e69c27b288b9209dcea9980: Status 404 returned error can't find the container with id a71babdb3341abd73a85d16b7a26da94d1b6565b9e69c27b288b9209dcea9980
Sep 09 12:49:49.943734 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.943688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2n2wj" event={"ID":"ab6539a3-0367-425b-9dc0-cc894772df3a","Type":"ContainerStarted","Data":"c6abea4ecc7481badec3cfd85260b9523ffb8880a9175afa0940493fd6a324a7"}
Sep 09 12:49:49.943930 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.943764    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:49:49.945705 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.945647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7jcs4" event={"ID":"2014baf8-52ee-420c-89d0-205989b32b87","Type":"ContainerStarted","Data":"a9b112da0acc2ec663c52c8277979546ac647e2a9a3d18a8c969cb44f7384d1b"}
Sep 09 12:49:49.947216 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.947171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6" event={"ID":"a65c5be9-4322-4a18-be47-6a49e215ad19","Type":"ContainerStarted","Data":"e8fad907f3139d4a861d52b27106ed9d41767f4344dfd0c1fd54395833de6923"}
Sep 09 12:49:49.949429 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.949400    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws" event={"ID":"8345b31d-f1bd-42ee-bf5e-b24dc175ce63","Type":"ContainerStarted","Data":"4d1ff4c6277be30cc74c0a6ac7269e889c6198f8957fe6f2d33cb71657a9d1cd"}
Sep 09 12:49:49.949535 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.949474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws" event={"ID":"8345b31d-f1bd-42ee-bf5e-b24dc175ce63","Type":"ContainerStarted","Data":"e58507a236a471b76626251b6fb30285f8489dcee00f44b01ccf3a37deebd536"}
Sep 09 12:49:49.950767 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.950742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qb94p" event={"ID":"53f9d380-aeec-4335-995d-5cd1df5e0026","Type":"ContainerStarted","Data":"baa3ac7556348ee4ef10c0ed91c9a1989ebc9600054845170c32e9e3cc19241a"}
Sep 09 12:49:49.952633 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.952604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerStarted","Data":"a71babdb3341abd73a85d16b7a26da94d1b6565b9e69c27b288b9209dcea9980"}
Sep 09 12:49:49.960900 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.960844    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2n2wj" podStartSLOduration=67.017523205 podStartE2EDuration="1m12.960807873s" totalImagesPullingTime="5.943284668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:49.95958854 +0000 UTC m=+72.879479879" watchObservedRunningTime="2026-09-09 12:49:49.960807873 +0000 UTC m=+72.880699211"
Sep 09 12:49:49.974897 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:49.974813    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7jcs4" podStartSLOduration=66.936801764 podStartE2EDuration="1m12.974795338s" totalImagesPullingTime="6.037993574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:49.974184396 +0000 UTC m=+72.894075735" watchObservedRunningTime="2026-09-09 12:49:49.974795338 +0000 UTC m=+72.894686676"
Sep 09 12:49:50.964121 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:50.963560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws" event={"ID":"8345b31d-f1bd-42ee-bf5e-b24dc175ce63","Type":"ContainerStarted","Data":"3f40c257dfd156e83aa5ee4dd83f9edbbc35ff1a3eccf8328e551243b17d393b"}
Sep 09 12:49:50.984219 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:50.984119    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-l5zws" podStartSLOduration=4.921365026 podStartE2EDuration="5.984093576s" totalImagesPullingTime="1.06272855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:50.982675712 +0000 UTC m=+73.902567091" watchObservedRunningTime="2026-09-09 12:49:50.984093576 +0000 UTC m=+73.903984928"
Sep 09 12:49:51.602351 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.600128    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:49:51.607149 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.604961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.608133 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.608109    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 12:49:51.612467 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.612437    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eogcbh283ndho\""
Sep 09 12:49:51.612690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.612666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 12:49:51.612863 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.612845    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 12:49:51.613536 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.613071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 12:49:51.613536 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.613093    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 12:49:51.613536 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.613290    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 12:49:51.613536 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.613302    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 12:49:51.613536 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.613316    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 12:49:51.613536 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.613529    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 12:49:51.613903 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.613748    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 12:49:51.613903 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.613836    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 12:49:51.614840 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.614102    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 12:49:51.614840 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.614281    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nn2q5\""
Sep 09 12:49:51.615004 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.614966    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 12:49:51.616490 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.616319    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 12:49:51.617573 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.617553    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 12:49:51.623353 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.623307    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:49:51.659553 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.659553 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.659553 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-config\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.659928 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.659928 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.659928 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.659928 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.659928 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.659928 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.659928 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.659928 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.660326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.659987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.660326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.660018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.660326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.660060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.660326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.660142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.660326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.660170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.660326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.660217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bksh\" (UniqueName: \"kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-kube-api-access-7bksh\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.660326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.660249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.660326 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.660323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.660709 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.660373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761061 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761263 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761263 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bksh\" (UniqueName: \"kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-kube-api-access-7bksh\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761263 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761263 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761263 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761263 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761263 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761604 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-config\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761604 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761604 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761604 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761604 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761604 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761604 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761604 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761604 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.761604 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.762010 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.762010 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.762010 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.761837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.762734 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.762592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.763212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.763184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.763410 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.763354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.764176 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.764153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.765305 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.764866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.765305 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.764993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.765606 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.765548    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.767298 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.767269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.767451 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.767428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.767767 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.767743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.768409 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.768391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.768544 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.768517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-config\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.770266 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.769981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.771032 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.770952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.771032 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.771006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.772281 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.772251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.772623 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.772571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.772914 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.772892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bksh\" (UniqueName: \"kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-kube-api-access-7bksh\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.775124 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.775098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:51.927226 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:51.927139    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:49:52.581197 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:52.581114    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:49:52.585759 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:49:52.585716    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c7c72a_be90_41e1_b393_8853a03f2878.slice/crio-8e4db14ac615df0071b29c8634cb6cd9bf46763bc607b47a3bf5dbe1aa19a0d0 WatchSource:0}: Error finding container 8e4db14ac615df0071b29c8634cb6cd9bf46763bc607b47a3bf5dbe1aa19a0d0: Status 404 returned error can't find the container with id 8e4db14ac615df0071b29c8634cb6cd9bf46763bc607b47a3bf5dbe1aa19a0d0
Sep 09 12:49:52.977400 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:52.977347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6" event={"ID":"a65c5be9-4322-4a18-be47-6a49e215ad19","Type":"ContainerStarted","Data":"9829ad1e1f02d46ff29c1cfff66e2567ba1db40061fa27c9bc6b6ba5b8ff9dcd"}
Sep 09 12:49:52.977593 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:52.977489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6" event={"ID":"a65c5be9-4322-4a18-be47-6a49e215ad19","Type":"ContainerStarted","Data":"3826fe9009f1179610f8485705be88d483a18775fc4cf9fda06fa5452d01868f"}
Sep 09 12:49:52.977593 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:52.977532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6" event={"ID":"a65c5be9-4322-4a18-be47-6a49e215ad19","Type":"ContainerStarted","Data":"9bc52b2f4a250f15ea2522949258a5e6ad096ebafdcc68a4a3a97ca63dcf0b83"}
Sep 09 12:49:52.979285 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:52.979254    2580 generic.go:356] "Generic (PLEG): container finished" podID="53f9d380-aeec-4335-995d-5cd1df5e0026" containerID="320cffaa9b32d5fac196152bd2e31369e6cfea062230a30e36c54983b9d35c78" exitCode=0
Sep 09 12:49:52.979411 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:52.979339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qb94p" event={"ID":"53f9d380-aeec-4335-995d-5cd1df5e0026","Type":"ContainerDied","Data":"320cffaa9b32d5fac196152bd2e31369e6cfea062230a30e36c54983b9d35c78"}
Sep 09 12:49:52.981143 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:52.981114    2580 generic.go:356] "Generic (PLEG): container finished" podID="29695e19-8030-4d99-82c1-56bdba6d3689" containerID="1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b" exitCode=0
Sep 09 12:49:52.981257 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:52.981183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerDied","Data":"1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b"}
Sep 09 12:49:52.983135 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:52.983110    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerID="59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3" exitCode=0
Sep 09 12:49:52.983243 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:52.983173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerDied","Data":"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"}
Sep 09 12:49:52.983243 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:52.983231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerStarted","Data":"8e4db14ac615df0071b29c8634cb6cd9bf46763bc607b47a3bf5dbe1aa19a0d0"}
Sep 09 12:49:53.996310 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:53.996028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6" event={"ID":"a65c5be9-4322-4a18-be47-6a49e215ad19","Type":"ContainerStarted","Data":"75f43dbcfe62a05f1aa9ea302e360bc5ba21d034de345dbe5a204f413de65ac9"}
Sep 09 12:49:53.996310 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:53.996163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6" event={"ID":"a65c5be9-4322-4a18-be47-6a49e215ad19","Type":"ContainerStarted","Data":"84644d20b7a7489574f2ca3287eeeb5c532542f46664259f48936563a3acdb17"}
Sep 09 12:49:53.996310 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:53.996211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6" event={"ID":"a65c5be9-4322-4a18-be47-6a49e215ad19","Type":"ContainerStarted","Data":"8d2192d93c2a21634156aec145f38d3faa37230bfdf0611c8968044ea63a18c6"}
Sep 09 12:49:53.996310 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:53.996261    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:49:54.000217 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:53.999641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qb94p" event={"ID":"53f9d380-aeec-4335-995d-5cd1df5e0026","Type":"ContainerStarted","Data":"cb3ed35e4010689e2cfa93c9fe197173163f58285f38b7043616ba73ee40e600"}
Sep 09 12:49:54.000217 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:53.999697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qb94p" event={"ID":"53f9d380-aeec-4335-995d-5cd1df5e0026","Type":"ContainerStarted","Data":"a552960bed634c6a0a9bc6f75d7e9b304851f01cad16003130390290b7ec9a77"}
Sep 09 12:49:54.020333 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:54.020275    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6" podStartSLOduration=3.081617342 podStartE2EDuration="7.020255803s" totalImagesPullingTime="3.938638461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:54.01850559 +0000 UTC m=+76.938396928" watchObservedRunningTime="2026-09-09 12:49:54.020255803 +0000 UTC m=+76.940147142"
Sep 09 12:49:54.050929 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:54.050836    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qb94p" podStartSLOduration=5.665634762 podStartE2EDuration="9.05078335s" totalImagesPullingTime="3.301727283s" totalInitContainerRuntime="83.421305ms" isStatefulPod=true podCreationTimestamp="2026-09-09 12:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:54.048798367 +0000 UTC m=+76.968689718" watchObservedRunningTime="2026-09-09 12:49:54.05078335 +0000 UTC m=+76.970674690"
Sep 09 12:49:54.857808 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:54.857664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b45d9f7f5-bzzcr"
Sep 09 12:49:55.009909 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:55.009176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerStarted","Data":"b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350"}
Sep 09 12:49:56.017594 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:56.017557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerStarted","Data":"10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c"}
Sep 09 12:49:56.018051 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:56.017680    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerStarted","Data":"2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df"}
Sep 09 12:49:56.018051 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:56.017727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerStarted","Data":"319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196"}
Sep 09 12:49:56.018051 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:56.017774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerStarted","Data":"8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6"}
Sep 09 12:49:56.018051 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:56.017813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerStarted","Data":"3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128"}
Sep 09 12:49:56.044963 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:49:56.044900    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.162015622 podStartE2EDuration="10.044879835s" totalImagesPullingTime="4.858997392s" totalInitContainerRuntime="23.866821ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:49:56.044618618 +0000 UTC m=+78.964509962" watchObservedRunningTime="2026-09-09 12:49:56.044879835 +0000 UTC m=+78.964771178"
Sep 09 12:50:00.015544 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:00.015412    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f4b64f9fb-gmdz6"
Sep 09 12:50:05.064938 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:05.064896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-mjfdh" event={"ID":"f09261ce-ec57-49c9-b05c-cdf587fc4853","Type":"ContainerStarted","Data":"84105532c6ce9a18a5127fb57d78e05c7a64c2febba53b1483bf2c8b1783e0cb"}
Sep 09 12:50:05.065376 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:05.065000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-mjfdh"
Sep 09 12:50:05.069412 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:05.069379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerStarted","Data":"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"}
Sep 09 12:50:05.069535 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:05.069451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerStarted","Data":"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"}
Sep 09 12:50:05.069535 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:05.069497    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerStarted","Data":"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"}
Sep 09 12:50:05.069535 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:05.069525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerStarted","Data":"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"}
Sep 09 12:50:05.069717 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:05.069555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerStarted","Data":"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"}
Sep 09 12:50:05.069717 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:05.069595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerStarted","Data":"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"}
Sep 09 12:50:05.078734 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:05.078709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-mjfdh"
Sep 09 12:50:05.087007 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:05.086956    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-mjfdh" podStartSLOduration=2.413052697 podStartE2EDuration="23.086938462s" totalImagesPullingTime="20.673885765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:50:05.084493904 +0000 UTC m=+88.004385243" watchObservedRunningTime="2026-09-09 12:50:05.086938462 +0000 UTC m=+88.006829801"
Sep 09 12:50:05.134918 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:05.134853    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.806113219 podStartE2EDuration="14.134799929s" totalImagesPullingTime="11.30118472s" totalInitContainerRuntime="27.50199ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:50:05.132955592 +0000 UTC m=+88.052846953" watchObservedRunningTime="2026-09-09 12:50:05.134799929 +0000 UTC m=+88.054691269"
Sep 09 12:50:06.927715 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:06.927675    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:50:20.966489 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:20.966459    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2n2wj"
Sep 09 12:50:27.687028 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:27.686995    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p8d9s_7faebb79-50d6-4521-b829-ba8c91fb8a9b/serve-healthcheck-canary/0.log"
Sep 09 12:50:51.927345 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:51.927305    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:50:51.948860 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:51.948827    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:50:52.283207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:50:52.283125    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:05.622376 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:05.622332    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:51:05.622926 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:05.622683    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="alertmanager" containerID="cri-o://b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350" gracePeriod=120
Sep 09 12:51:05.622926 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:05.622760    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy-metric" containerID="cri-o://2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df" gracePeriod=120
Sep 09 12:51:05.622926 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:05.622801    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy-web" containerID="cri-o://8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6" gracePeriod=120
Sep 09 12:51:05.622926 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:05.622880    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="config-reloader" containerID="cri-o://3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128" gracePeriod=120
Sep 09 12:51:05.623154 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:05.622934    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy" containerID="cri-o://319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196" gracePeriod=120
Sep 09 12:51:05.623154 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:05.622808    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="prom-label-proxy" containerID="cri-o://10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c" gracePeriod=120
Sep 09 12:51:06.323590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.323553    2580 generic.go:356] "Generic (PLEG): container finished" podID="29695e19-8030-4d99-82c1-56bdba6d3689" containerID="10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c" exitCode=0
Sep 09 12:51:06.323590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.323580    2580 generic.go:356] "Generic (PLEG): container finished" podID="29695e19-8030-4d99-82c1-56bdba6d3689" containerID="319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196" exitCode=0
Sep 09 12:51:06.323590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.323587    2580 generic.go:356] "Generic (PLEG): container finished" podID="29695e19-8030-4d99-82c1-56bdba6d3689" containerID="3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128" exitCode=0
Sep 09 12:51:06.323590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.323593    2580 generic.go:356] "Generic (PLEG): container finished" podID="29695e19-8030-4d99-82c1-56bdba6d3689" containerID="b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350" exitCode=0
Sep 09 12:51:06.323879 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.323623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerDied","Data":"10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c"}
Sep 09 12:51:06.323879 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.323708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerDied","Data":"319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196"}
Sep 09 12:51:06.323879 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.323740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerDied","Data":"3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128"}
Sep 09 12:51:06.323879 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.323764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerDied","Data":"b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350"}
Sep 09 12:51:06.869094 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.869069    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:06.958100 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958071    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958299 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958128    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-trusted-ca-bundle\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958299 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-metric\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958299 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958274    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-config-out\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958299 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958292    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-main-db\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958547 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958333    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-cluster-tls-config\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958547 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-web\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958547 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-metrics-client-ca\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958547 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-config-volume\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958547 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958461    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-tls-assets\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958547 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958492    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-web-config\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958547 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:51:06.958547 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958523    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958972 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgv5j\" (UniqueName: \"kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-kube-api-access-kgv5j\") pod \"29695e19-8030-4d99-82c1-56bdba6d3689\" (UID: \"29695e19-8030-4d99-82c1-56bdba6d3689\") "
Sep 09 12:51:06.958972 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:51:06.958972 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958869    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:06.958972 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.958893    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-alertmanager-main-db\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:06.959693 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.959658    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:51:06.961440 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.961409    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:06.963538 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.963301    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-config-out" (OuterVolumeSpecName: "config-out") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:51:06.963538 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.963315    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:51:06.963538 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.963370    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:06.963538 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.963468    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-kube-api-access-kgv5j" (OuterVolumeSpecName: "kube-api-access-kgv5j") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "kube-api-access-kgv5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:51:06.963538 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.963514    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-config-volume" (OuterVolumeSpecName: "config-volume") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:06.963895 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.963729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:06.964701 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.964676    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:06.967897 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.967872    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:06.974874 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:06.974850    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-web-config" (OuterVolumeSpecName: "web-config") pod "29695e19-8030-4d99-82c1-56bdba6d3689" (UID: "29695e19-8030-4d99-82c1-56bdba6d3689"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:07.060121 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.060083    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-main-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:07.060121 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.060112    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgv5j\" (UniqueName: \"kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-kube-api-access-kgv5j\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:07.060121 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.060123    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:07.060399 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.060134    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:07.060399 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.060144    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29695e19-8030-4d99-82c1-56bdba6d3689-config-out\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:07.060399 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.060151    2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-cluster-tls-config\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:07.060399 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.060160    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:07.060399 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.060168    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29695e19-8030-4d99-82c1-56bdba6d3689-metrics-client-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:07.060399 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.060176    2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:07.060399 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.060184    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29695e19-8030-4d99-82c1-56bdba6d3689-tls-assets\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:07.060399 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.060193    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29695e19-8030-4d99-82c1-56bdba6d3689-web-config\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:07.331298 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.331262    2580 generic.go:356] "Generic (PLEG): container finished" podID="29695e19-8030-4d99-82c1-56bdba6d3689" containerID="2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df" exitCode=0
Sep 09 12:51:07.331298 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.331289    2580 generic.go:356] "Generic (PLEG): container finished" podID="29695e19-8030-4d99-82c1-56bdba6d3689" containerID="8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6" exitCode=0
Sep 09 12:51:07.331499 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.331311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerDied","Data":"2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df"}
Sep 09 12:51:07.331499 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.331348    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.331499 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.331371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerDied","Data":"8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6"}
Sep 09 12:51:07.331499 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.331398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29695e19-8030-4d99-82c1-56bdba6d3689","Type":"ContainerDied","Data":"a71babdb3341abd73a85d16b7a26da94d1b6565b9e69c27b288b9209dcea9980"}
Sep 09 12:51:07.331499 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.331428    2580 scope.go:117] "RemoveContainer" containerID="10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c"
Sep 09 12:51:07.339879 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.339860    2580 scope.go:117] "RemoveContainer" containerID="2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df"
Sep 09 12:51:07.347139 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.347124    2580 scope.go:117] "RemoveContainer" containerID="319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196"
Sep 09 12:51:07.354601 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.354581    2580 scope.go:117] "RemoveContainer" containerID="8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6"
Sep 09 12:51:07.355803 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.355787    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:51:07.360186 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.360168    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:51:07.362770 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.362752    2580 scope.go:117] "RemoveContainer" containerID="3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128"
Sep 09 12:51:07.369853 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.369838    2580 scope.go:117] "RemoveContainer" containerID="b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350"
Sep 09 12:51:07.377195 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.377175    2580 scope.go:117] "RemoveContainer" containerID="1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b"
Sep 09 12:51:07.384116 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.384097    2580 scope.go:117] "RemoveContainer" containerID="10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c"
Sep 09 12:51:07.384728 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:07.384566    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c\": container with ID starting with 10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c not found: ID does not exist" containerID="10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c"
Sep 09 12:51:07.384728 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.384605    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c"} err="failed to get container status \"10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c\": rpc error: code = NotFound desc = could not find container \"10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c\": container with ID starting with 10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c not found: ID does not exist"
Sep 09 12:51:07.384728 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.384650    2580 scope.go:117] "RemoveContainer" containerID="2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df"
Sep 09 12:51:07.385057 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:07.385033    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df\": container with ID starting with 2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df not found: ID does not exist" containerID="2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df"
Sep 09 12:51:07.385154 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.385066    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df"} err="failed to get container status \"2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df\": rpc error: code = NotFound desc = could not find container \"2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df\": container with ID starting with 2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df not found: ID does not exist"
Sep 09 12:51:07.385154 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.385091    2580 scope.go:117] "RemoveContainer" containerID="319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196"
Sep 09 12:51:07.385343 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:07.385327    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196\": container with ID starting with 319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196 not found: ID does not exist" containerID="319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196"
Sep 09 12:51:07.385398 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.385349    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196"} err="failed to get container status \"319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196\": rpc error: code = NotFound desc = could not find container \"319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196\": container with ID starting with 319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196 not found: ID does not exist"
Sep 09 12:51:07.385398 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.385369    2580 scope.go:117] "RemoveContainer" containerID="8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6"
Sep 09 12:51:07.385652 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:07.385635    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6\": container with ID starting with 8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6 not found: ID does not exist" containerID="8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6"
Sep 09 12:51:07.385700 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.385657    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6"} err="failed to get container status \"8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6\": rpc error: code = NotFound desc = could not find container \"8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6\": container with ID starting with 8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6 not found: ID does not exist"
Sep 09 12:51:07.385700 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.385672    2580 scope.go:117] "RemoveContainer" containerID="3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128"
Sep 09 12:51:07.385956 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:07.385936    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128\": container with ID starting with 3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128 not found: ID does not exist" containerID="3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128"
Sep 09 12:51:07.386032 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.385963    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128"} err="failed to get container status \"3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128\": rpc error: code = NotFound desc = could not find container \"3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128\": container with ID starting with 3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128 not found: ID does not exist"
Sep 09 12:51:07.386032 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.385982    2580 scope.go:117] "RemoveContainer" containerID="b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350"
Sep 09 12:51:07.386307 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:07.386289    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350\": container with ID starting with b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350 not found: ID does not exist" containerID="b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350"
Sep 09 12:51:07.386380 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.386313    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350"} err="failed to get container status \"b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350\": rpc error: code = NotFound desc = could not find container \"b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350\": container with ID starting with b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350 not found: ID does not exist"
Sep 09 12:51:07.386380 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.386332    2580 scope.go:117] "RemoveContainer" containerID="1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b"
Sep 09 12:51:07.386623 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:07.386598    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b\": container with ID starting with 1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b not found: ID does not exist" containerID="1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b"
Sep 09 12:51:07.386668 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.386618    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b"} err="failed to get container status \"1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b\": rpc error: code = NotFound desc = could not find container \"1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b\": container with ID starting with 1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b not found: ID does not exist"
Sep 09 12:51:07.386668 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.386632    2580 scope.go:117] "RemoveContainer" containerID="10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c"
Sep 09 12:51:07.386911 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.386891    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c"} err="failed to get container status \"10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c\": rpc error: code = NotFound desc = could not find container \"10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c\": container with ID starting with 10e52d0f04a9be509674b594f430691639112198c39440624ea6eb9289b12c0c not found: ID does not exist"
Sep 09 12:51:07.386959 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.386917    2580 scope.go:117] "RemoveContainer" containerID="2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df"
Sep 09 12:51:07.386959 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.386941    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:51:07.387224 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387206    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df"} err="failed to get container status \"2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df\": rpc error: code = NotFound desc = could not find container \"2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df\": container with ID starting with 2d29c6c0e678411fd7c808c2ea51671e7e5dec95194ff7253bde3af912df00df not found: ID does not exist"
Sep 09 12:51:07.387299 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387225    2580 scope.go:117] "RemoveContainer" containerID="319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196"
Sep 09 12:51:07.387299 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387278    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="config-reloader"
Sep 09 12:51:07.387299 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387291    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="config-reloader"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387311    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="prom-label-proxy"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387317    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="prom-label-proxy"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387325    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387331    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387337    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy-web"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387342    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy-web"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387352    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy-metric"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387359    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy-metric"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387364    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="init-config-reloader"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387372    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="init-config-reloader"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387385    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="alertmanager"
Sep 09 12:51:07.387416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387391    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="alertmanager"
Sep 09 12:51:07.387790 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387467    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy"
Sep 09 12:51:07.387790 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387475    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy-metric"
Sep 09 12:51:07.387790 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387469    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196"} err="failed to get container status \"319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196\": rpc error: code = NotFound desc = could not find container \"319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196\": container with ID starting with 319781baafbcdd7c407e48f90fd75514355bbbb0bafa3a2eea7cc642ffb75196 not found: ID does not exist"
Sep 09 12:51:07.387790 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387499    2580 scope.go:117] "RemoveContainer" containerID="8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6"
Sep 09 12:51:07.387790 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387482    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="config-reloader"
Sep 09 12:51:07.387790 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387590    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="kube-rbac-proxy-web"
Sep 09 12:51:07.387790 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387605    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="alertmanager"
Sep 09 12:51:07.387790 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387616    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" containerName="prom-label-proxy"
Sep 09 12:51:07.388144 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387836    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6"} err="failed to get container status \"8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6\": rpc error: code = NotFound desc = could not find container \"8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6\": container with ID starting with 8c38f1a2d6018186346c0b38b4b63abf88e6f7504bb014d8a1d8d291ed1d8ee6 not found: ID does not exist"
Sep 09 12:51:07.388144 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.387861    2580 scope.go:117] "RemoveContainer" containerID="3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128"
Sep 09 12:51:07.388219 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.388167    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128"} err="failed to get container status \"3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128\": rpc error: code = NotFound desc = could not find container \"3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128\": container with ID starting with 3ff29e2deb082e37306c129b227c4d2d8f9877229115834fb665e2fdfba29128 not found: ID does not exist"
Sep 09 12:51:07.388219 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.388185    2580 scope.go:117] "RemoveContainer" containerID="b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350"
Sep 09 12:51:07.389064 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.388972    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350"} err="failed to get container status \"b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350\": rpc error: code = NotFound desc = could not find container \"b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350\": container with ID starting with b28d324fc8513297208a9eb4239774e64f84a16daf9455b4e876eb44cc3e1350 not found: ID does not exist"
Sep 09 12:51:07.389064 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.389001    2580 scope.go:117] "RemoveContainer" containerID="1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b"
Sep 09 12:51:07.389491 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.389465    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b"} err="failed to get container status \"1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b\": rpc error: code = NotFound desc = could not find container \"1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b\": container with ID starting with 1c14236c99483481da5faf3213302dd2234682c0457d4c44dbea398b5db6050b not found: ID does not exist"
Sep 09 12:51:07.391892 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.391874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.394724 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.394695    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 12:51:07.394724 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.394720    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 12:51:07.394920 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.394695    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 12:51:07.394920 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.394697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-26v94\""
Sep 09 12:51:07.394920 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.394697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 12:51:07.395184 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.395164    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 12:51:07.395359 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.395169    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 12:51:07.395359 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.395294    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 12:51:07.395359 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.395295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 12:51:07.399654 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.399575    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 12:51:07.401917 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.401892    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:51:07.463196 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-config-volume\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463196 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463409 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9322d04f-d442-45ac-8932-122cb25bcadd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463409 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9322d04f-d442-45ac-8932-122cb25bcadd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463409 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463409 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9322d04f-d442-45ac-8932-122cb25bcadd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463409 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463559 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-web-config\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463559 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463559 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463559 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml6jd\" (UniqueName: \"kubernetes.io/projected/9322d04f-d442-45ac-8932-122cb25bcadd-kube-api-access-ml6jd\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463559 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9322d04f-d442-45ac-8932-122cb25bcadd-config-out\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.463703 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.463567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9322d04f-d442-45ac-8932-122cb25bcadd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.563991 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.563963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-config-volume\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.563991 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.563994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.564226 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9322d04f-d442-45ac-8932-122cb25bcadd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.564226 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9322d04f-d442-45ac-8932-122cb25bcadd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.564226 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.564226 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9322d04f-d442-45ac-8932-122cb25bcadd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.564226 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.564226 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-web-config\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.564226 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.564447 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.564447 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml6jd\" (UniqueName: \"kubernetes.io/projected/9322d04f-d442-45ac-8932-122cb25bcadd-kube-api-access-ml6jd\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.564447 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9322d04f-d442-45ac-8932-122cb25bcadd-config-out\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.564447 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9322d04f-d442-45ac-8932-122cb25bcadd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.565235 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.564910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9322d04f-d442-45ac-8932-122cb25bcadd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.565235 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.565197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9322d04f-d442-45ac-8932-122cb25bcadd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.565456 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.565283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9322d04f-d442-45ac-8932-122cb25bcadd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.567361 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.567305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.567543 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.567486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-config-volume\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.568182 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.568157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.568299 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.568279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.568498 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.568439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9322d04f-d442-45ac-8932-122cb25bcadd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.568717 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.568680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.568787 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.568735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-web-config\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.569456 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.569436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9322d04f-d442-45ac-8932-122cb25bcadd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.569852 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.569811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9322d04f-d442-45ac-8932-122cb25bcadd-config-out\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.572283 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.572266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml6jd\" (UniqueName: \"kubernetes.io/projected/9322d04f-d442-45ac-8932-122cb25bcadd-kube-api-access-ml6jd\") pod \"alertmanager-main-0\" (UID: \"9322d04f-d442-45ac-8932-122cb25bcadd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.595742 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.595677    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29695e19-8030-4d99-82c1-56bdba6d3689" path="/var/lib/kubelet/pods/29695e19-8030-4d99-82c1-56bdba6d3689/volumes"
Sep 09 12:51:07.702601 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.702553    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:51:07.982416 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:07.982394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:51:07.984790 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:51:07.984766    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9322d04f_d442_45ac_8932_122cb25bcadd.slice/crio-54b4aef1828535d9d6a8fdc003fac9361d418bcaf14977674e7c3fc5d53733dc WatchSource:0}: Error finding container 54b4aef1828535d9d6a8fdc003fac9361d418bcaf14977674e7c3fc5d53733dc: Status 404 returned error can't find the container with id 54b4aef1828535d9d6a8fdc003fac9361d418bcaf14977674e7c3fc5d53733dc
Sep 09 12:51:08.337627 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:08.337596    2580 generic.go:356] "Generic (PLEG): container finished" podID="9322d04f-d442-45ac-8932-122cb25bcadd" containerID="4e08628619215e55e5995f3ce881642f0cb4eb880b065c0f5515da5f7aa455bb" exitCode=0
Sep 09 12:51:08.337796 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:08.337683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9322d04f-d442-45ac-8932-122cb25bcadd","Type":"ContainerDied","Data":"4e08628619215e55e5995f3ce881642f0cb4eb880b065c0f5515da5f7aa455bb"}
Sep 09 12:51:08.337796 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:08.337771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9322d04f-d442-45ac-8932-122cb25bcadd","Type":"ContainerStarted","Data":"54b4aef1828535d9d6a8fdc003fac9361d418bcaf14977674e7c3fc5d53733dc"}
Sep 09 12:51:09.345565 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.345529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9322d04f-d442-45ac-8932-122cb25bcadd","Type":"ContainerStarted","Data":"9ae257c0debb05175a20594dd42d40e7cdbd49106756876cc4ca5ee712e4f40c"}
Sep 09 12:51:09.345987 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.345597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9322d04f-d442-45ac-8932-122cb25bcadd","Type":"ContainerStarted","Data":"d9351385c4aba64761d5c582b2a360e376005375599829ea3f3307510ccde1f2"}
Sep 09 12:51:09.345987 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.345624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9322d04f-d442-45ac-8932-122cb25bcadd","Type":"ContainerStarted","Data":"66d09b67bb50e64fcac2b8b80c6534f63128c928a2adf5a2cd7aca93042f93b7"}
Sep 09 12:51:09.345987 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.345648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9322d04f-d442-45ac-8932-122cb25bcadd","Type":"ContainerStarted","Data":"0000a1dcf1e67eac260aa2a3a29906eafe7269a66306d92f898db542f57569b0"}
Sep 09 12:51:09.345987 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.345674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9322d04f-d442-45ac-8932-122cb25bcadd","Type":"ContainerStarted","Data":"20ba16358a8f6f1f4fa2c8b6906ac40c4ebfc33f081cacb8d55136aa391a841c"}
Sep 09 12:51:09.345987 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.345698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9322d04f-d442-45ac-8932-122cb25bcadd","Type":"ContainerStarted","Data":"90c0a2dd57d22254704aa83541a3eb991a6c1bc7a5e1566943e4c03ac4ef34a0"}
Sep 09 12:51:09.372932 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.372874    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.355936586 podStartE2EDuration="2.372853739s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.917153ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:51:09.371248915 +0000 UTC m=+152.291140255" watchObservedRunningTime="2026-09-09 12:51:09.372853739 +0000 UTC m=+152.292745079"
Sep 09 12:51:09.654131 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.654030    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-58666c69f4-294bn"]
Sep 09 12:51:09.656691 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.656667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.661161 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.660967    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 12:51:09.661161 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.661000    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 12:51:09.661161 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.661005    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 12:51:09.661520 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.661295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 12:51:09.661520 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.661420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qmrbv\""
Sep 09 12:51:09.661520 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.661454    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 12:51:09.664613 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.664588    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 12:51:09.670711 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.670687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58666c69f4-294bn"]
Sep 09 12:51:09.785892 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.785857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43408383-4017-450f-98ba-356093595129-serving-certs-ca-bundle\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.785892 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.785909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-federate-client-tls\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.786138 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.785982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.786138 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.786042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-telemeter-client-tls\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.786138 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.786061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43408383-4017-450f-98ba-356093595129-metrics-client-ca\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.786138 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.786092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjr2\" (UniqueName: \"kubernetes.io/projected/43408383-4017-450f-98ba-356093595129-kube-api-access-9wjr2\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.786324 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.786210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43408383-4017-450f-98ba-356093595129-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.786324 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.786251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-secret-telemeter-client\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.887281 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.887242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjr2\" (UniqueName: \"kubernetes.io/projected/43408383-4017-450f-98ba-356093595129-kube-api-access-9wjr2\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.887457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.887302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43408383-4017-450f-98ba-356093595129-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.887457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.887318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-secret-telemeter-client\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.887457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.887339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43408383-4017-450f-98ba-356093595129-serving-certs-ca-bundle\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.887457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.887365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-federate-client-tls\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.887457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.887392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.887457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.887413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-telemeter-client-tls\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.887457 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.887433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43408383-4017-450f-98ba-356093595129-metrics-client-ca\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.888304 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.888270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43408383-4017-450f-98ba-356093595129-serving-certs-ca-bundle\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.888304 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.888297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43408383-4017-450f-98ba-356093595129-metrics-client-ca\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.888634 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.888299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43408383-4017-450f-98ba-356093595129-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.890368 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.890339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.890475 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.890347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-telemeter-client-tls\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.890558 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.890544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-federate-client-tls\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.890675 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.890654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/43408383-4017-450f-98ba-356093595129-secret-telemeter-client\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.895487 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.895465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjr2\" (UniqueName: \"kubernetes.io/projected/43408383-4017-450f-98ba-356093595129-kube-api-access-9wjr2\") pod \"telemeter-client-58666c69f4-294bn\" (UID: \"43408383-4017-450f-98ba-356093595129\") " pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:09.960771 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.960675    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:51:09.962789 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.961076    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="prometheus" containerID="cri-o://63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686" gracePeriod=600
Sep 09 12:51:09.962789 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.961273    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy-web" containerID="cri-o://a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c" gracePeriod=600
Sep 09 12:51:09.962789 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.961274    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy" containerID="cri-o://c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85" gracePeriod=600
Sep 09 12:51:09.962789 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.961374    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="thanos-sidecar" containerID="cri-o://2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52" gracePeriod=600
Sep 09 12:51:09.962789 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.961442    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy-thanos" containerID="cri-o://042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724" gracePeriod=600
Sep 09 12:51:09.962789 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.961446    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="config-reloader" containerID="cri-o://14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39" gracePeriod=600
Sep 09 12:51:09.970539 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:09.970434    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58666c69f4-294bn"
Sep 09 12:51:10.213777 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.213711    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.271523 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.270500    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58666c69f4-294bn"]
Sep 09 12:51:10.273362 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:51:10.273333    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43408383_4017_450f_98ba_356093595129.slice/crio-fd8ad8dc1d3390c823f8ff599e75d76e6f2ef110c03e72bcd52add73d0204e9f WatchSource:0}: Error finding container fd8ad8dc1d3390c823f8ff599e75d76e6f2ef110c03e72bcd52add73d0204e9f: Status 404 returned error can't find the container with id fd8ad8dc1d3390c823f8ff599e75d76e6f2ef110c03e72bcd52add73d0204e9f
Sep 09 12:51:10.291243 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-config-out\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291402 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bksh\" (UniqueName: \"kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-kube-api-access-7bksh\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291402 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291303    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-metrics-client-certs\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291402 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291330    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291584 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291472    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-web-config\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291633 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-1\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291633 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-trusted-ca-bundle\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291736 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291660    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-kube-rbac-proxy\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291736 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291736 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291715    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-0\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291903 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291754    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-serving-certs-ca-bundle\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291903 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291794    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-grpc-tls\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291903 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-config\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.291903 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291879    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-2\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.292081 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-thanos-prometheus-http-client-file\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.292081 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291952    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-tls\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.292081 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.291997    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-db\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.292081 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.292023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-metrics-client-ca\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.292081 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.292074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-tls-assets\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.292303 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.292099    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-kubelet-serving-ca-bundle\") pod \"f6c7c72a-be90-41e1-b393-8853a03f2878\" (UID: \"f6c7c72a-be90-41e1-b393-8853a03f2878\") "
Sep 09 12:51:10.294288 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.292541    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:51:10.294288 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.292777    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:51:10.294288 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.293973    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:51:10.294580 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.294478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:51:10.294580 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.294567    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:51:10.295418 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.294885    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:51:10.295418 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.295206    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:51:10.295418 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.295308    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:51:10.295638 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.295618    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:10.296258 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.296176    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:10.296258 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.296226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-config-out" (OuterVolumeSpecName: "config-out") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:51:10.296258 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.296239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-kube-api-access-7bksh" (OuterVolumeSpecName: "kube-api-access-7bksh") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "kube-api-access-7bksh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:51:10.297092 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.297052    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:10.297948 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.297907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:10.298235 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.298199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-config" (OuterVolumeSpecName: "config") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:10.299248 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.298609    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:10.299781 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.299744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:10.301038 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.301011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:51:10.301124 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.301047    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:10.311859 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.311805    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-web-config" (OuterVolumeSpecName: "web-config") pod "f6c7c72a-be90-41e1-b393-8853a03f2878" (UID: "f6c7c72a-be90-41e1-b393-8853a03f2878"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:10.353510 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353476    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerID="042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724" exitCode=0
Sep 09 12:51:10.353510 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353503    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerID="c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85" exitCode=0
Sep 09 12:51:10.353510 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353510    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerID="a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c" exitCode=0
Sep 09 12:51:10.353510 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353515    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerID="2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52" exitCode=0
Sep 09 12:51:10.353510 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353519    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerID="14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39" exitCode=0
Sep 09 12:51:10.354149 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353525    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerID="63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686" exitCode=0
Sep 09 12:51:10.354149 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353561    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.354149 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerDied","Data":"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"}
Sep 09 12:51:10.354149 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerDied","Data":"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"}
Sep 09 12:51:10.354149 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353652    2580 scope.go:117] "RemoveContainer" containerID="042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"
Sep 09 12:51:10.354149 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerDied","Data":"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"}
Sep 09 12:51:10.354149 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerDied","Data":"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"}
Sep 09 12:51:10.354149 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerDied","Data":"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"}
Sep 09 12:51:10.354149 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerDied","Data":"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"}
Sep 09 12:51:10.354149 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.353797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6c7c72a-be90-41e1-b393-8853a03f2878","Type":"ContainerDied","Data":"8e4db14ac615df0071b29c8634cb6cd9bf46763bc607b47a3bf5dbe1aa19a0d0"}
Sep 09 12:51:10.355174 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.355147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58666c69f4-294bn" event={"ID":"43408383-4017-450f-98ba-356093595129","Type":"ContainerStarted","Data":"fd8ad8dc1d3390c823f8ff599e75d76e6f2ef110c03e72bcd52add73d0204e9f"}
Sep 09 12:51:10.362539 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.362516    2580 scope.go:117] "RemoveContainer" containerID="c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"
Sep 09 12:51:10.370710 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.370690    2580 scope.go:117] "RemoveContainer" containerID="a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"
Sep 09 12:51:10.378579 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.378557    2580 scope.go:117] "RemoveContainer" containerID="2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"
Sep 09 12:51:10.378644 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.378556    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:51:10.383583 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.383564    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:51:10.386317 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.386302    2580 scope.go:117] "RemoveContainer" containerID="14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"
Sep 09 12:51:10.392783 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392764    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-metrics-client-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392783 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392783    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392794    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-web-config\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392804    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392812    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392840    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-kube-rbac-proxy\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392851    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392860    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392869    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392877    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-grpc-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392885    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-config\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392892    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392901    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392910    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6c7c72a-be90-41e1-b393-8853a03f2878-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392918    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-prometheus-k8s-db\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392926    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-metrics-client-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392933    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-tls-assets\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392942    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6c7c72a-be90-41e1-b393-8853a03f2878-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392962    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6c7c72a-be90-41e1-b393-8853a03f2878-config-out\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.392988 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.392971    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bksh\" (UniqueName: \"kubernetes.io/projected/f6c7c72a-be90-41e1-b393-8853a03f2878-kube-api-access-7bksh\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 12:51:10.393905 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.393769    2580 scope.go:117] "RemoveContainer" containerID="63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"
Sep 09 12:51:10.401306 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.401288    2580 scope.go:117] "RemoveContainer" containerID="59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"
Sep 09 12:51:10.408507 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.408469    2580 scope.go:117] "RemoveContainer" containerID="042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"
Sep 09 12:51:10.408926 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:10.408840    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": container with ID starting with 042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724 not found: ID does not exist" containerID="042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"
Sep 09 12:51:10.408926 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.408878    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"} err="failed to get container status \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": rpc error: code = NotFound desc = could not find container \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": container with ID starting with 042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724 not found: ID does not exist"
Sep 09 12:51:10.408926 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.408905    2580 scope.go:117] "RemoveContainer" containerID="c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"
Sep 09 12:51:10.409239 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:10.409216    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": container with ID starting with c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85 not found: ID does not exist" containerID="c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"
Sep 09 12:51:10.409317 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.409248    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"} err="failed to get container status \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": rpc error: code = NotFound desc = could not find container \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": container with ID starting with c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85 not found: ID does not exist"
Sep 09 12:51:10.409317 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.409267    2580 scope.go:117] "RemoveContainer" containerID="a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"
Sep 09 12:51:10.409544 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:10.409527    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": container with ID starting with a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c not found: ID does not exist" containerID="a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"
Sep 09 12:51:10.409590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.409549    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"} err="failed to get container status \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": rpc error: code = NotFound desc = could not find container \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": container with ID starting with a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c not found: ID does not exist"
Sep 09 12:51:10.409590 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.409562    2580 scope.go:117] "RemoveContainer" containerID="2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"
Sep 09 12:51:10.409810 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:10.409794    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": container with ID starting with 2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52 not found: ID does not exist" containerID="2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"
Sep 09 12:51:10.409905 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.409812    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"} err="failed to get container status \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": rpc error: code = NotFound desc = could not find container \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": container with ID starting with 2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52 not found: ID does not exist"
Sep 09 12:51:10.409905 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.409855    2580 scope.go:117] "RemoveContainer" containerID="14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"
Sep 09 12:51:10.410108 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:10.410093    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": container with ID starting with 14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39 not found: ID does not exist" containerID="14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"
Sep 09 12:51:10.410164 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410110    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"} err="failed to get container status \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": rpc error: code = NotFound desc = could not find container \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": container with ID starting with 14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39 not found: ID does not exist"
Sep 09 12:51:10.410164 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410122    2580 scope.go:117] "RemoveContainer" containerID="63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"
Sep 09 12:51:10.410164 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410140    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:51:10.410387 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:10.410369    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": container with ID starting with 63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686 not found: ID does not exist" containerID="63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"
Sep 09 12:51:10.410421 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410392    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"} err="failed to get container status \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": rpc error: code = NotFound desc = could not find container \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": container with ID starting with 63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686 not found: ID does not exist"
Sep 09 12:51:10.410421 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410406    2580 scope.go:117] "RemoveContainer" containerID="59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"
Sep 09 12:51:10.410521 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410508    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy-web"
Sep 09 12:51:10.410561 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410524    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy-web"
Sep 09 12:51:10.410561 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410539    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="init-config-reloader"
Sep 09 12:51:10.410561 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410548    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="init-config-reloader"
Sep 09 12:51:10.410561 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410558    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy-thanos"
Sep 09 12:51:10.410690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410564    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy-thanos"
Sep 09 12:51:10.410690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410578    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="thanos-sidecar"
Sep 09 12:51:10.410690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410585    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="thanos-sidecar"
Sep 09 12:51:10.410690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410597    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="prometheus"
Sep 09 12:51:10.410690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410603    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="prometheus"
Sep 09 12:51:10.410690 ip-10-0-140-133 kubenswrapper[2580]: E0909 12:51:10.410606    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": container with ID starting with 59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3 not found: ID does not exist" containerID="59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"
Sep 09 12:51:10.410690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410626    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"} err="failed to get container status \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": rpc error: code = NotFound desc = could not find container \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": container with ID starting with 59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3 not found: ID does not exist"
Sep 09 12:51:10.410690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410640    2580 scope.go:117] "RemoveContainer" containerID="042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"
Sep 09 12:51:10.410690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410610    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="config-reloader"
Sep 09 12:51:10.410690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410682    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="config-reloader"
Sep 09 12:51:10.410690 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410693    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy"
Sep 09 12:51:10.411085 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410698    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy"
Sep 09 12:51:10.411085 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410773    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy-thanos"
Sep 09 12:51:10.411085 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410787    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy-web"
Sep 09 12:51:10.411085 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410797    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="config-reloader"
Sep 09 12:51:10.411085 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410806    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="prometheus"
Sep 09 12:51:10.411085 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410831    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="thanos-sidecar"
Sep 09 12:51:10.411085 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410839    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" containerName="kube-rbac-proxy"
Sep 09 12:51:10.411085 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410906    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"} err="failed to get container status \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": rpc error: code = NotFound desc = could not find container \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": container with ID starting with 042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724 not found: ID does not exist"
Sep 09 12:51:10.411085 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.410931    2580 scope.go:117] "RemoveContainer" containerID="c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"
Sep 09 12:51:10.411375 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.411175    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"} err="failed to get container status \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": rpc error: code = NotFound desc = could not find container \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": container with ID starting with c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85 not found: ID does not exist"
Sep 09 12:51:10.411375 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.411189    2580 scope.go:117] "RemoveContainer" containerID="a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"
Sep 09 12:51:10.411496 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.411383    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"} err="failed to get container status \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": rpc error: code = NotFound desc = could not find container \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": container with ID starting with a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c not found: ID does not exist"
Sep 09 12:51:10.411496 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.411397    2580 scope.go:117] "RemoveContainer" containerID="2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"
Sep 09 12:51:10.411633 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.411612    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"} err="failed to get container status \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": rpc error: code = NotFound desc = could not find container \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": container with ID starting with 2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52 not found: ID does not exist"
Sep 09 12:51:10.411677 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.411634    2580 scope.go:117] "RemoveContainer" containerID="14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"
Sep 09 12:51:10.411851 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.411835    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"} err="failed to get container status \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": rpc error: code = NotFound desc = could not find container \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": container with ID starting with 14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39 not found: ID does not exist"
Sep 09 12:51:10.411904 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.411851    2580 scope.go:117] "RemoveContainer" containerID="63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"
Sep 09 12:51:10.412060 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.412045    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"} err="failed to get container status \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": rpc error: code = NotFound desc = could not find container \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": container with ID starting with 63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686 not found: ID does not exist"
Sep 09 12:51:10.412101 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.412061    2580 scope.go:117] "RemoveContainer" containerID="59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"
Sep 09 12:51:10.412266 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.412251    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"} err="failed to get container status \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": rpc error: code = NotFound desc = could not find container \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": container with ID starting with 59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3 not found: ID does not exist"
Sep 09 12:51:10.412310 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.412266    2580 scope.go:117] "RemoveContainer" containerID="042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"
Sep 09 12:51:10.412491 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.412476    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"} err="failed to get container status \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": rpc error: code = NotFound desc = could not find container \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": container with ID starting with 042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724 not found: ID does not exist"
Sep 09 12:51:10.412491 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.412490    2580 scope.go:117] "RemoveContainer" containerID="c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"
Sep 09 12:51:10.412699 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.412679    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"} err="failed to get container status \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": rpc error: code = NotFound desc = could not find container \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": container with ID starting with c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85 not found: ID does not exist"
Sep 09 12:51:10.412739 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.412700    2580 scope.go:117] "RemoveContainer" containerID="a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"
Sep 09 12:51:10.412972 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.412955    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"} err="failed to get container status \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": rpc error: code = NotFound desc = could not find container \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": container with ID starting with a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c not found: ID does not exist"
Sep 09 12:51:10.413023 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.412972    2580 scope.go:117] "RemoveContainer" containerID="2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"
Sep 09 12:51:10.413193 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.413177    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"} err="failed to get container status \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": rpc error: code = NotFound desc = could not find container \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": container with ID starting with 2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52 not found: ID does not exist"
Sep 09 12:51:10.413193 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.413193    2580 scope.go:117] "RemoveContainer" containerID="14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"
Sep 09 12:51:10.413393 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.413377    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"} err="failed to get container status \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": rpc error: code = NotFound desc = could not find container \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": container with ID starting with 14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39 not found: ID does not exist"
Sep 09 12:51:10.413432 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.413393    2580 scope.go:117] "RemoveContainer" containerID="63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"
Sep 09 12:51:10.413605 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.413584    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"} err="failed to get container status \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": rpc error: code = NotFound desc = could not find container \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": container with ID starting with 63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686 not found: ID does not exist"
Sep 09 12:51:10.413605 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.413601    2580 scope.go:117] "RemoveContainer" containerID="59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"
Sep 09 12:51:10.413855 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.413801    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"} err="failed to get container status \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": rpc error: code = NotFound desc = could not find container \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": container with ID starting with 59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3 not found: ID does not exist"
Sep 09 12:51:10.413945 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.413882    2580 scope.go:117] "RemoveContainer" containerID="042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"
Sep 09 12:51:10.414093 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.414075    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"} err="failed to get container status \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": rpc error: code = NotFound desc = could not find container \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": container with ID starting with 042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724 not found: ID does not exist"
Sep 09 12:51:10.414285 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.414093    2580 scope.go:117] "RemoveContainer" containerID="c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"
Sep 09 12:51:10.414459 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.414288    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"} err="failed to get container status \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": rpc error: code = NotFound desc = could not find container \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": container with ID starting with c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85 not found: ID does not exist"
Sep 09 12:51:10.414459 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.414314    2580 scope.go:117] "RemoveContainer" containerID="a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"
Sep 09 12:51:10.414606 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.414581    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"} err="failed to get container status \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": rpc error: code = NotFound desc = could not find container \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": container with ID starting with a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c not found: ID does not exist"
Sep 09 12:51:10.414670 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.414606    2580 scope.go:117] "RemoveContainer" containerID="2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"
Sep 09 12:51:10.414927 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.414907    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"} err="failed to get container status \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": rpc error: code = NotFound desc = could not find container \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": container with ID starting with 2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52 not found: ID does not exist"
Sep 09 12:51:10.415011 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.414928    2580 scope.go:117] "RemoveContainer" containerID="14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"
Sep 09 12:51:10.415187 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.415161    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"} err="failed to get container status \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": rpc error: code = NotFound desc = could not find container \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": container with ID starting with 14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39 not found: ID does not exist"
Sep 09 12:51:10.415187 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.415181    2580 scope.go:117] "RemoveContainer" containerID="63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"
Sep 09 12:51:10.415281 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.415240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.415830 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.415449    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"} err="failed to get container status \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": rpc error: code = NotFound desc = could not find container \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": container with ID starting with 63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686 not found: ID does not exist"
Sep 09 12:51:10.415830 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.415476    2580 scope.go:117] "RemoveContainer" containerID="59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"
Sep 09 12:51:10.415983 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.415924    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"} err="failed to get container status \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": rpc error: code = NotFound desc = could not find container \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": container with ID starting with 59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3 not found: ID does not exist"
Sep 09 12:51:10.415983 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.415942    2580 scope.go:117] "RemoveContainer" containerID="042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"
Sep 09 12:51:10.416208 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.416180    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"} err="failed to get container status \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": rpc error: code = NotFound desc = could not find container \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": container with ID starting with 042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724 not found: ID does not exist"
Sep 09 12:51:10.416208 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.416205    2580 scope.go:117] "RemoveContainer" containerID="c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"
Sep 09 12:51:10.416477 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.416459    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"} err="failed to get container status \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": rpc error: code = NotFound desc = could not find container \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": container with ID starting with c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85 not found: ID does not exist"
Sep 09 12:51:10.416477 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.416477    2580 scope.go:117] "RemoveContainer" containerID="a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"
Sep 09 12:51:10.416757 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.416739    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"} err="failed to get container status \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": rpc error: code = NotFound desc = could not find container \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": container with ID starting with a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c not found: ID does not exist"
Sep 09 12:51:10.416881 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.416757    2580 scope.go:117] "RemoveContainer" containerID="2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"
Sep 09 12:51:10.417074 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.417055    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"} err="failed to get container status \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": rpc error: code = NotFound desc = could not find container \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": container with ID starting with 2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52 not found: ID does not exist"
Sep 09 12:51:10.417115 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.417075    2580 scope.go:117] "RemoveContainer" containerID="14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"
Sep 09 12:51:10.417340 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.417322    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"} err="failed to get container status \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": rpc error: code = NotFound desc = could not find container \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": container with ID starting with 14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39 not found: ID does not exist"
Sep 09 12:51:10.417390 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.417340    2580 scope.go:117] "RemoveContainer" containerID="63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"
Sep 09 12:51:10.417593 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.417579    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"} err="failed to get container status \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": rpc error: code = NotFound desc = could not find container \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": container with ID starting with 63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686 not found: ID does not exist"
Sep 09 12:51:10.417637 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.417594    2580 scope.go:117] "RemoveContainer" containerID="59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"
Sep 09 12:51:10.417864 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.417847    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"} err="failed to get container status \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": rpc error: code = NotFound desc = could not find container \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": container with ID starting with 59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3 not found: ID does not exist"
Sep 09 12:51:10.417864 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.417864    2580 scope.go:117] "RemoveContainer" containerID="042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"
Sep 09 12:51:10.418124 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.418104    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724"} err="failed to get container status \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": rpc error: code = NotFound desc = could not find container \"042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724\": container with ID starting with 042aed697c0117a2e4cfad5df73351c0ded7730efc0ade4fb22f92fe753a3724 not found: ID does not exist"
Sep 09 12:51:10.418191 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.418126    2580 scope.go:117] "RemoveContainer" containerID="c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"
Sep 09 12:51:10.418415 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.418384    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85"} err="failed to get container status \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": rpc error: code = NotFound desc = could not find container \"c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85\": container with ID starting with c0c97df43556254636a25ffbe31e4d2b3ea8729aa145bc9e481aded59c98ba85 not found: ID does not exist"
Sep 09 12:51:10.418495 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.418416    2580 scope.go:117] "RemoveContainer" containerID="a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"
Sep 09 12:51:10.418495 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.418421    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 12:51:10.418713 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.418669    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c"} err="failed to get container status \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": rpc error: code = NotFound desc = could not find container \"a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c\": container with ID starting with a5deb9fb7933fd97bd687987cc85e0dacec1d454ed4e9a214bf3affdf1e8f83c not found: ID does not exist"
Sep 09 12:51:10.418713 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.418693    2580 scope.go:117] "RemoveContainer" containerID="2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"
Sep 09 12:51:10.418845 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.418715    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 12:51:10.418845 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.418737    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nn2q5\""
Sep 09 12:51:10.418845 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.418679    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 12:51:10.419116 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419094    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52"} err="failed to get container status \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": rpc error: code = NotFound desc = could not find container \"2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52\": container with ID starting with 2b003f51584dd491b0ba593a6670d991b0ad105e0a59242f5fb56612ffe71f52 not found: ID does not exist"
Sep 09 12:51:10.419188 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419117    2580 scope.go:117] "RemoveContainer" containerID="14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"
Sep 09 12:51:10.419188 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419147    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 12:51:10.419188 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419173    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 12:51:10.419371 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419348    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39"} err="failed to get container status \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": rpc error: code = NotFound desc = could not find container \"14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39\": container with ID starting with 14f43914ea796f6b5647d9b3d1ec9e6618f2c23129ce7ab0db2bce1febdc8a39 not found: ID does not exist"
Sep 09 12:51:10.419436 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419374    2580 scope.go:117] "RemoveContainer" containerID="63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"
Sep 09 12:51:10.419436 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419403    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eogcbh283ndho\""
Sep 09 12:51:10.419541 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 12:51:10.419591 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419537    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 12:51:10.419645 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419591    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686"} err="failed to get container status \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": rpc error: code = NotFound desc = could not find container \"63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686\": container with ID starting with 63a8270ea75c2ee8f69c8e7016724f310c9cc66a34f113ef0e404d7e9ce12686 not found: ID does not exist"
Sep 09 12:51:10.419645 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419612    2580 scope.go:117] "RemoveContainer" containerID="59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"
Sep 09 12:51:10.419759 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419670    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 12:51:10.419759 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419738    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 12:51:10.419873 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419792    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 12:51:10.420002 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.419981    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3"} err="failed to get container status \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": rpc error: code = NotFound desc = could not find container \"59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3\": container with ID starting with 59b006d15b8c678a55ff18da87e40fed954c4e0bd2be6180ce0fbccb5c7017b3 not found: ID does not exist"
Sep 09 12:51:10.420278 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.420259    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 12:51:10.420341 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.420276    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 12:51:10.420516 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.420501    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 12:51:10.421756 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.421689    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 12:51:10.428582 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.424688    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 12:51:10.432496 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.432469    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:51:10.493842 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.493735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-config-out\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494024 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.493998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494106 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nwc\" (UniqueName: \"kubernetes.io/projected/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-kube-api-access-l7nwc\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494106 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494218 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-web-config\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494218 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494218 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494320 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494320 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494408 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494477 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494512 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494564 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494526    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494626 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494685 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494733 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494786 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494786 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494889 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-config\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.494889 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.494864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.595830 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.595787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.595999 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.595863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.595999 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.595897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-config\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.595999 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.595921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.595999 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.595952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-config-out\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596213 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596213 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nwc\" (UniqueName: \"kubernetes.io/projected/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-kube-api-access-l7nwc\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596213 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596213 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-web-config\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596403 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596553 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596609 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596660 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596910 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596910 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596910 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596910 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596910 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596910 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596910 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.596910 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.597300 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.597300 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.596922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.598042 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.597681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.599711 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.599112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-config-out\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.599711 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.599333    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.599711 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.599363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-config\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.599711 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.599652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.600035 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.599872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.600035 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.599911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.600138 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.600080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-web-config\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.600882 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.600676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.600882 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.600800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.601057 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.600903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.601124 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.601104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.602153 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.602126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.602242 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.602157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.602242 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.602216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.602242 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.602223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.604770 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.604740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nwc\" (UniqueName: \"kubernetes.io/projected/58b06d01-fb3f-4ca8-85f9-ef2732628b1f-kube-api-access-l7nwc\") pod \"prometheus-k8s-0\" (UID: \"58b06d01-fb3f-4ca8-85f9-ef2732628b1f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:10.731813 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:10.731774    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:51:11.015068 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:11.014959    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:51:11.019058 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:51:11.019023    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b06d01_fb3f_4ca8_85f9_ef2732628b1f.slice/crio-c02773dbe621de4a2b74418b1a660ed62b16f5039c27f2ecc8b965af4f9e8b0a WatchSource:0}: Error finding container c02773dbe621de4a2b74418b1a660ed62b16f5039c27f2ecc8b965af4f9e8b0a: Status 404 returned error can't find the container with id c02773dbe621de4a2b74418b1a660ed62b16f5039c27f2ecc8b965af4f9e8b0a
Sep 09 12:51:11.363013 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:11.362925    2580 generic.go:356] "Generic (PLEG): container finished" podID="58b06d01-fb3f-4ca8-85f9-ef2732628b1f" containerID="31c67ad53ae9eeae5f06d0331d86fabe61c0ca2816a237163ff57d4e31aaa90c" exitCode=0
Sep 09 12:51:11.363013 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:11.362998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"58b06d01-fb3f-4ca8-85f9-ef2732628b1f","Type":"ContainerDied","Data":"31c67ad53ae9eeae5f06d0331d86fabe61c0ca2816a237163ff57d4e31aaa90c"}
Sep 09 12:51:11.363548 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:11.363073    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"58b06d01-fb3f-4ca8-85f9-ef2732628b1f","Type":"ContainerStarted","Data":"c02773dbe621de4a2b74418b1a660ed62b16f5039c27f2ecc8b965af4f9e8b0a"}
Sep 09 12:51:11.596893 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:11.596149    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6c7c72a-be90-41e1-b393-8853a03f2878" path="/var/lib/kubelet/pods/f6c7c72a-be90-41e1-b393-8853a03f2878/volumes"
Sep 09 12:51:12.371667 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:12.371628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58666c69f4-294bn" event={"ID":"43408383-4017-450f-98ba-356093595129","Type":"ContainerStarted","Data":"1304d89099ae0e767204f52666e85d1a332651e8260cbedd098745a9589a19ac"}
Sep 09 12:51:12.372105 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:12.371698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58666c69f4-294bn" event={"ID":"43408383-4017-450f-98ba-356093595129","Type":"ContainerStarted","Data":"aa698d3fd14257906897ba0a2830d3a7bd4c3e8f1c85a8a71dd35c8aaa75a5de"}
Sep 09 12:51:12.372105 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:12.371723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58666c69f4-294bn" event={"ID":"43408383-4017-450f-98ba-356093595129","Type":"ContainerStarted","Data":"7fc225a14202ad7863106c7c170c821221b2c495b6b2b21644a94b2e67b9df78"}
Sep 09 12:51:12.375179 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:12.375104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"58b06d01-fb3f-4ca8-85f9-ef2732628b1f","Type":"ContainerStarted","Data":"fde5286bb548cf217ec025519d3d3cdd330626de45ff683f870580d69ddd605c"}
Sep 09 12:51:12.375179 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:12.375170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"58b06d01-fb3f-4ca8-85f9-ef2732628b1f","Type":"ContainerStarted","Data":"b8de76bd5ffcf37596b44e93559314233e06564fc8945631257d235b95bed1db"}
Sep 09 12:51:12.375362 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:12.375206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"58b06d01-fb3f-4ca8-85f9-ef2732628b1f","Type":"ContainerStarted","Data":"fcda4b88e71230b8a09a71b27699e3d821c8abf002728c25e9aa60a5bd8f5e28"}
Sep 09 12:51:12.375362 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:12.375238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"58b06d01-fb3f-4ca8-85f9-ef2732628b1f","Type":"ContainerStarted","Data":"d304ff9cb0f8ddfa820435c46e1ee2e17180c2224de45eb32aca60ad957e31a7"}
Sep 09 12:51:12.375362 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:12.375267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"58b06d01-fb3f-4ca8-85f9-ef2732628b1f","Type":"ContainerStarted","Data":"865f0c0439865d0504a227a143a772818d8669d0b8a220e222e8d927e81feb6e"}
Sep 09 12:51:12.375362 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:12.375289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"58b06d01-fb3f-4ca8-85f9-ef2732628b1f","Type":"ContainerStarted","Data":"c20e72dd717efcbf31a8addb7fa8aea828bd79e6740fe6b5cb5c4b1de1b0e2ef"}
Sep 09 12:51:12.398757 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:12.398280    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-58666c69f4-294bn" podStartSLOduration=1.745824295 podStartE2EDuration="3.398258107s" totalImagesPullingTime="1.652433812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:51:12.395649682 +0000 UTC m=+155.315541033" watchObservedRunningTime="2026-09-09 12:51:12.398258107 +0000 UTC m=+155.318149446"
Sep 09 12:51:12.427533 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:12.427424    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.404350403 podStartE2EDuration="2.427406557s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.056154ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:51:12.425513674 +0000 UTC m=+155.345405014" watchObservedRunningTime="2026-09-09 12:51:12.427406557 +0000 UTC m=+155.347297896"
Sep 09 12:51:15.732144 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:51:15.732104    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:52:10.732596 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:52:10.732556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:52:10.749174 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:52:10.749145    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:52:11.634224 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:52:11.634186    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:53:37.476183 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:53:37.476151    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 12:53:37.481010 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:53:37.480990    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 12:53:37.482595 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:53:37.482577    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 12:54:52.209594 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.209519    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"]
Sep 09 12:54:52.212880 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.212864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.216294 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.216271    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 12:54:52.217209 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.217193    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 12:54:52.217469 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.217456    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-g9kqz\""
Sep 09 12:54:52.217520 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.217493    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 12:54:52.217571 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.217518    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:54:52.217919 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.217903    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 12:54:52.225886 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.225867    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"]
Sep 09 12:54:52.317242 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.317199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc9f0746-4d40-443c-b859-615ffb404eb3-metrics-cert\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.317242 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.317240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxns5\" (UniqueName: \"kubernetes.io/projected/bc9f0746-4d40-443c-b859-615ffb404eb3-kube-api-access-sxns5\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.317470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.317346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc9f0746-4d40-443c-b859-615ffb404eb3-cert\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.317470 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.317378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc9f0746-4d40-443c-b859-615ffb404eb3-manager-config\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.418071 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.418041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc9f0746-4d40-443c-b859-615ffb404eb3-cert\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.418071 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.418074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc9f0746-4d40-443c-b859-615ffb404eb3-manager-config\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.418288 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.418124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc9f0746-4d40-443c-b859-615ffb404eb3-metrics-cert\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.418288 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.418141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxns5\" (UniqueName: \"kubernetes.io/projected/bc9f0746-4d40-443c-b859-615ffb404eb3-kube-api-access-sxns5\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.418779 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.418756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc9f0746-4d40-443c-b859-615ffb404eb3-manager-config\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.420655 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.420638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc9f0746-4d40-443c-b859-615ffb404eb3-cert\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.420711 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.420681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc9f0746-4d40-443c-b859-615ffb404eb3-metrics-cert\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.427886 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.427861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxns5\" (UniqueName: \"kubernetes.io/projected/bc9f0746-4d40-443c-b859-615ffb404eb3-kube-api-access-sxns5\") pod \"lws-controller-manager-798b96659f-zpsks\" (UID: \"bc9f0746-4d40-443c-b859-615ffb404eb3\") " pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.522350 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.522263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:54:52.793260 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.793237    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"]
Sep 09 12:54:52.795481 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:54:52.795455    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9f0746_4d40_443c_b859_615ffb404eb3.slice/crio-1dc23318712e8080c53f93503ad1717c9ab1661d3bb24b088f1e23340bdcd721 WatchSource:0}: Error finding container 1dc23318712e8080c53f93503ad1717c9ab1661d3bb24b088f1e23340bdcd721: Status 404 returned error can't find the container with id 1dc23318712e8080c53f93503ad1717c9ab1661d3bb24b088f1e23340bdcd721
Sep 09 12:54:52.797516 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:52.797500    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:54:53.244664 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:54:53.244632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks" event={"ID":"bc9f0746-4d40-443c-b859-615ffb404eb3","Type":"ContainerStarted","Data":"1dc23318712e8080c53f93503ad1717c9ab1661d3bb24b088f1e23340bdcd721"}
Sep 09 12:55:04.001400 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.001372    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"]
Sep 09 12:55:04.004703 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.004686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.010319 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.010294    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 12:55:04.010486 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.010459    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 12:55:04.010547 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.010507    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vtl8s\""
Sep 09 12:55:04.010998 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.010979    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 12:55:04.011109 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.011004    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 12:55:04.024265 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.024243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"]
Sep 09 12:55:04.115864 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.115840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/540d3089-efae-4516-a6e1-c70d67f08eec-tmp\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.115969 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.115870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/540d3089-efae-4516-a6e1-c70d67f08eec-webhook-cert\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.115969 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.115899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqhkf\" (UniqueName: \"kubernetes.io/projected/540d3089-efae-4516-a6e1-c70d67f08eec-kube-api-access-cqhkf\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.115969 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.115959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/540d3089-efae-4516-a6e1-c70d67f08eec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.116084 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.116006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/540d3089-efae-4516-a6e1-c70d67f08eec-manifests\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.217160 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.217132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/540d3089-efae-4516-a6e1-c70d67f08eec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.217267 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.217182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/540d3089-efae-4516-a6e1-c70d67f08eec-manifests\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.217267 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.217220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/540d3089-efae-4516-a6e1-c70d67f08eec-tmp\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.217267 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.217237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/540d3089-efae-4516-a6e1-c70d67f08eec-webhook-cert\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.217398 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.217274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqhkf\" (UniqueName: \"kubernetes.io/projected/540d3089-efae-4516-a6e1-c70d67f08eec-kube-api-access-cqhkf\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.217668 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.217636    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/540d3089-efae-4516-a6e1-c70d67f08eec-manifests\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.219642 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.219623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/540d3089-efae-4516-a6e1-c70d67f08eec-tmp\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.219879 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.219862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/540d3089-efae-4516-a6e1-c70d67f08eec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.219951 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.219900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/540d3089-efae-4516-a6e1-c70d67f08eec-webhook-cert\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.228583 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.228558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqhkf\" (UniqueName: \"kubernetes.io/projected/540d3089-efae-4516-a6e1-c70d67f08eec-kube-api-access-cqhkf\") pod \"opendatahub-operator-controller-manager-647774d94-2wgx4\" (UID: \"540d3089-efae-4516-a6e1-c70d67f08eec\") " pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.315579 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.315559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:04.598439 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:04.598411    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"]
Sep 09 12:55:04.602413 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:55:04.602388    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540d3089_efae_4516_a6e1_c70d67f08eec.slice/crio-70901f465b42af38f2d5093258020fba89d630a47a1a9f6c1235e6f6b36ec8b5 WatchSource:0}: Error finding container 70901f465b42af38f2d5093258020fba89d630a47a1a9f6c1235e6f6b36ec8b5: Status 404 returned error can't find the container with id 70901f465b42af38f2d5093258020fba89d630a47a1a9f6c1235e6f6b36ec8b5
Sep 09 12:55:05.297647 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:05.297602    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4" event={"ID":"540d3089-efae-4516-a6e1-c70d67f08eec","Type":"ContainerStarted","Data":"70901f465b42af38f2d5093258020fba89d630a47a1a9f6c1235e6f6b36ec8b5"}
Sep 09 12:55:09.319893 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:09.319851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4" event={"ID":"540d3089-efae-4516-a6e1-c70d67f08eec","Type":"ContainerStarted","Data":"40a0de416b74c2e12997bd85582f93c555331a987d98c0fbb594dab4f2374334"}
Sep 09 12:55:09.321458 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:09.321431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks" event={"ID":"bc9f0746-4d40-443c-b859-615ffb404eb3","Type":"ContainerStarted","Data":"2d291a02eef7c54e31cab894d5b21a8942936aa83c0c69c04d8037b2c343368e"}
Sep 09 12:55:09.321549 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:09.321488    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:55:09.355787 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:09.355745    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks" podStartSLOduration=1.4050948939999999 podStartE2EDuration="17.355731076s" totalImagesPullingTime="15.950636182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:55:09.355123672 +0000 UTC m=+392.275015012" watchObservedRunningTime="2026-09-09 12:55:09.355731076 +0000 UTC m=+392.275622415"
Sep 09 12:55:10.327032 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:10.326998    2580 generic.go:356] "Generic (PLEG): container finished" podID="540d3089-efae-4516-a6e1-c70d67f08eec" containerID="40a0de416b74c2e12997bd85582f93c555331a987d98c0fbb594dab4f2374334" exitCode=0
Sep 09 12:55:10.327408 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:10.327055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4" event={"ID":"540d3089-efae-4516-a6e1-c70d67f08eec","Type":"ContainerDied","Data":"40a0de416b74c2e12997bd85582f93c555331a987d98c0fbb594dab4f2374334"}
Sep 09 12:55:11.334163 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:11.334122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4" event={"ID":"540d3089-efae-4516-a6e1-c70d67f08eec","Type":"ContainerStarted","Data":"a89eb6351f2584f4505d206c59a88b762aa4741477bb2856a8214f5fa9ae0504"}
Sep 09 12:55:11.334607 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:11.334275    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:11.369122 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:11.369069    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4" podStartSLOduration=2.784844033 podStartE2EDuration="8.369054095s" totalImagesPullingTime="4.502702399s" totalInitContainerRuntime="1.081507663s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:55:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:55:11.367448425 +0000 UTC m=+394.287339764" watchObservedRunningTime="2026-09-09 12:55:11.369054095 +0000 UTC m=+394.288945435"
Sep 09 12:55:20.329090 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:20.329058    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-798b96659f-zpsks"
Sep 09 12:55:22.340475 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:22.340447    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-647774d94-2wgx4"
Sep 09 12:55:24.443287 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.443258    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-768599446d-qthzs"]
Sep 09 12:55:24.520239 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.520208    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-768599446d-qthzs"]
Sep 09 12:55:24.520388 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.520302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs"
Sep 09 12:55:24.527117 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.527094    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 12:55:24.527251 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.527094    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vjgfv\""
Sep 09 12:55:24.527251 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.527096    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 12:55:24.527251 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.527165    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 12:55:24.527251 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.527200    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 12:55:24.573654 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.573632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c5c960-e0c7-4df3-9a77-c9516507f8ec-tls-certs\") pod \"kube-auth-proxy-768599446d-qthzs\" (UID: \"38c5c960-e0c7-4df3-9a77-c9516507f8ec\") " pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs"
Sep 09 12:55:24.573762 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.573655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38c5c960-e0c7-4df3-9a77-c9516507f8ec-tmp\") pod \"kube-auth-proxy-768599446d-qthzs\" (UID: \"38c5c960-e0c7-4df3-9a77-c9516507f8ec\") " pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs"
Sep 09 12:55:24.573855 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.573755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kctf7\" (UniqueName: \"kubernetes.io/projected/38c5c960-e0c7-4df3-9a77-c9516507f8ec-kube-api-access-kctf7\") pod \"kube-auth-proxy-768599446d-qthzs\" (UID: \"38c5c960-e0c7-4df3-9a77-c9516507f8ec\") " pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs"
Sep 09 12:55:24.674223 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.674187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c5c960-e0c7-4df3-9a77-c9516507f8ec-tls-certs\") pod \"kube-auth-proxy-768599446d-qthzs\" (UID: \"38c5c960-e0c7-4df3-9a77-c9516507f8ec\") " pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs"
Sep 09 12:55:24.674223 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.674223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38c5c960-e0c7-4df3-9a77-c9516507f8ec-tmp\") pod \"kube-auth-proxy-768599446d-qthzs\" (UID: \"38c5c960-e0c7-4df3-9a77-c9516507f8ec\") " pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs"
Sep 09 12:55:24.674391 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.674345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kctf7\" (UniqueName: \"kubernetes.io/projected/38c5c960-e0c7-4df3-9a77-c9516507f8ec-kube-api-access-kctf7\") pod \"kube-auth-proxy-768599446d-qthzs\" (UID: \"38c5c960-e0c7-4df3-9a77-c9516507f8ec\") " pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs"
Sep 09 12:55:24.676513 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.676485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38c5c960-e0c7-4df3-9a77-c9516507f8ec-tmp\") pod \"kube-auth-proxy-768599446d-qthzs\" (UID: \"38c5c960-e0c7-4df3-9a77-c9516507f8ec\") " pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs"
Sep 09 12:55:24.676597 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.676581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c5c960-e0c7-4df3-9a77-c9516507f8ec-tls-certs\") pod \"kube-auth-proxy-768599446d-qthzs\" (UID: \"38c5c960-e0c7-4df3-9a77-c9516507f8ec\") " pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs"
Sep 09 12:55:24.682639 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.682614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kctf7\" (UniqueName: \"kubernetes.io/projected/38c5c960-e0c7-4df3-9a77-c9516507f8ec-kube-api-access-kctf7\") pod \"kube-auth-proxy-768599446d-qthzs\" (UID: \"38c5c960-e0c7-4df3-9a77-c9516507f8ec\") " pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs"
Sep 09 12:55:24.829674 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:24.829644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs"
Sep 09 12:55:25.136677 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:25.136654    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-768599446d-qthzs"]
Sep 09 12:55:25.139115 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:55:25.139086    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c5c960_e0c7_4df3_9a77_c9516507f8ec.slice/crio-2203779b2e05498d1f2f5444648f54236aeec029127abda72b4a740c09b74cf4 WatchSource:0}: Error finding container 2203779b2e05498d1f2f5444648f54236aeec029127abda72b4a740c09b74cf4: Status 404 returned error can't find the container with id 2203779b2e05498d1f2f5444648f54236aeec029127abda72b4a740c09b74cf4
Sep 09 12:55:25.395883 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:25.395805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs" event={"ID":"38c5c960-e0c7-4df3-9a77-c9516507f8ec","Type":"ContainerStarted","Data":"2203779b2e05498d1f2f5444648f54236aeec029127abda72b4a740c09b74cf4"}
Sep 09 12:55:30.423650 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:30.423610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs" event={"ID":"38c5c960-e0c7-4df3-9a77-c9516507f8ec","Type":"ContainerStarted","Data":"6e3d25400d663f47ddfc5e4191c0172113c5c0daffe14dbd3c3077f072dfd330"}
Sep 09 12:55:30.439872 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:55:30.439808    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-768599446d-qthzs" podStartSLOduration=1.710415329 podStartE2EDuration="6.439794004s" totalImagesPullingTime="4.729378675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:55:30.438613213 +0000 UTC m=+413.358504552" watchObservedRunningTime="2026-09-09 12:55:30.439794004 +0000 UTC m=+413.359685342"
Sep 09 12:57:09.367748 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.367706    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"]
Sep 09 12:57:09.370655 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.370637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"
Sep 09 12:57:09.373137 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.373104    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 12:57:09.373238 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.373176    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kq5kw\""
Sep 09 12:57:09.373289 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.373269    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 12:57:09.374207 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.374190    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 12:57:09.374276 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.374225    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 12:57:09.381386 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.381367    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"]
Sep 09 12:57:09.525184 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.525151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/297fb6db-f5f9-4711-abcc-1640a374b64f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dwc5b\" (UID: \"297fb6db-f5f9-4711-abcc-1640a374b64f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"
Sep 09 12:57:09.525331 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.525192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65c9h\" (UniqueName: \"kubernetes.io/projected/297fb6db-f5f9-4711-abcc-1640a374b64f-kube-api-access-65c9h\") pod \"kuadrant-console-plugin-fbdd4f445-dwc5b\" (UID: \"297fb6db-f5f9-4711-abcc-1640a374b64f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"
Sep 09 12:57:09.525331 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.525320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/297fb6db-f5f9-4711-abcc-1640a374b64f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dwc5b\" (UID: \"297fb6db-f5f9-4711-abcc-1640a374b64f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"
Sep 09 12:57:09.626346 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.626273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/297fb6db-f5f9-4711-abcc-1640a374b64f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dwc5b\" (UID: \"297fb6db-f5f9-4711-abcc-1640a374b64f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"
Sep 09 12:57:09.626346 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.626333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/297fb6db-f5f9-4711-abcc-1640a374b64f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dwc5b\" (UID: \"297fb6db-f5f9-4711-abcc-1640a374b64f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"
Sep 09 12:57:09.626493 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.626353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65c9h\" (UniqueName: \"kubernetes.io/projected/297fb6db-f5f9-4711-abcc-1640a374b64f-kube-api-access-65c9h\") pod \"kuadrant-console-plugin-fbdd4f445-dwc5b\" (UID: \"297fb6db-f5f9-4711-abcc-1640a374b64f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"
Sep 09 12:57:09.626936 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.626906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/297fb6db-f5f9-4711-abcc-1640a374b64f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dwc5b\" (UID: \"297fb6db-f5f9-4711-abcc-1640a374b64f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"
Sep 09 12:57:09.628914 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.628895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/297fb6db-f5f9-4711-abcc-1640a374b64f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dwc5b\" (UID: \"297fb6db-f5f9-4711-abcc-1640a374b64f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"
Sep 09 12:57:09.644782 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.644760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65c9h\" (UniqueName: \"kubernetes.io/projected/297fb6db-f5f9-4711-abcc-1640a374b64f-kube-api-access-65c9h\") pod \"kuadrant-console-plugin-fbdd4f445-dwc5b\" (UID: \"297fb6db-f5f9-4711-abcc-1640a374b64f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"
Sep 09 12:57:09.679165 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.679145    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"
Sep 09 12:57:09.943264 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:09.943240    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b"]
Sep 09 12:57:09.946356 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:57:09.946325    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod297fb6db_f5f9_4711_abcc_1640a374b64f.slice/crio-3af0059645838e431af9dbf6e56b43b8f7b173595f9a6bfa8ab1dc0c59b273eb WatchSource:0}: Error finding container 3af0059645838e431af9dbf6e56b43b8f7b173595f9a6bfa8ab1dc0c59b273eb: Status 404 returned error can't find the container with id 3af0059645838e431af9dbf6e56b43b8f7b173595f9a6bfa8ab1dc0c59b273eb
Sep 09 12:57:10.855104 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:10.855063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b" event={"ID":"297fb6db-f5f9-4711-abcc-1640a374b64f","Type":"ContainerStarted","Data":"3af0059645838e431af9dbf6e56b43b8f7b173595f9a6bfa8ab1dc0c59b273eb"}
Sep 09 12:57:14.876903 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:14.876792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b" event={"ID":"297fb6db-f5f9-4711-abcc-1640a374b64f","Type":"ContainerStarted","Data":"5170581824f4ab7dadcc20e69c56f8e8d03ac7c27ed8a2d6054e44ed2dbf7f82"}
Sep 09 12:57:14.891649 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:57:14.891599    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dwc5b" podStartSLOduration=1.2556398610000001 podStartE2EDuration="5.891585139s" totalImagesPullingTime="4.635945278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:57:14.891396002 +0000 UTC m=+517.811287344" watchObservedRunningTime="2026-09-09 12:57:14.891585139 +0000 UTC m=+517.811476524"
Sep 09 12:58:04.475075 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.474997    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 12:58:04.478212 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.478195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"
Sep 09 12:58:04.483522 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.483499    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 12:58:04.499372 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.499346    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 12:58:04.539898 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.539874    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 12:58:04.589960 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.589932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdglc\" (UniqueName: \"kubernetes.io/projected/60a102b1-adad-4daa-a957-c8a4eba449a1-kube-api-access-bdglc\") pod \"limitador-limitador-84994f7ffc-wh5m6\" (UID: \"60a102b1-adad-4daa-a957-c8a4eba449a1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"
Sep 09 12:58:04.590073 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.590003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60a102b1-adad-4daa-a957-c8a4eba449a1-config-file\") pod \"limitador-limitador-84994f7ffc-wh5m6\" (UID: \"60a102b1-adad-4daa-a957-c8a4eba449a1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"
Sep 09 12:58:04.691404 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.691371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdglc\" (UniqueName: \"kubernetes.io/projected/60a102b1-adad-4daa-a957-c8a4eba449a1-kube-api-access-bdglc\") pod \"limitador-limitador-84994f7ffc-wh5m6\" (UID: \"60a102b1-adad-4daa-a957-c8a4eba449a1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"
Sep 09 12:58:04.691566 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.691429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60a102b1-adad-4daa-a957-c8a4eba449a1-config-file\") pod \"limitador-limitador-84994f7ffc-wh5m6\" (UID: \"60a102b1-adad-4daa-a957-c8a4eba449a1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"
Sep 09 12:58:04.691998 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.691981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60a102b1-adad-4daa-a957-c8a4eba449a1-config-file\") pod \"limitador-limitador-84994f7ffc-wh5m6\" (UID: \"60a102b1-adad-4daa-a957-c8a4eba449a1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"
Sep 09 12:58:04.700340 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.700320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdglc\" (UniqueName: \"kubernetes.io/projected/60a102b1-adad-4daa-a957-c8a4eba449a1-kube-api-access-bdglc\") pod \"limitador-limitador-84994f7ffc-wh5m6\" (UID: \"60a102b1-adad-4daa-a957-c8a4eba449a1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"
Sep 09 12:58:04.788371 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:04.788308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"
Sep 09 12:58:05.069635 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:05.069611    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 12:58:05.071905 ip-10-0-140-133 kubenswrapper[2580]: W0909 12:58:05.071879    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a102b1_adad_4daa_a957_c8a4eba449a1.slice/crio-b59be7ae7e8f37d09db505de83180c1bf4671fd86d1d2512a2b724a6c4e187cd WatchSource:0}: Error finding container b59be7ae7e8f37d09db505de83180c1bf4671fd86d1d2512a2b724a6c4e187cd: Status 404 returned error can't find the container with id b59be7ae7e8f37d09db505de83180c1bf4671fd86d1d2512a2b724a6c4e187cd
Sep 09 12:58:05.101101 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:05.101075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6" event={"ID":"60a102b1-adad-4daa-a957-c8a4eba449a1","Type":"ContainerStarted","Data":"b59be7ae7e8f37d09db505de83180c1bf4671fd86d1d2512a2b724a6c4e187cd"}
Sep 09 12:58:08.119024 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:08.118992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6" event={"ID":"60a102b1-adad-4daa-a957-c8a4eba449a1","Type":"ContainerStarted","Data":"7a4d4b9291343579c68fa6885c75ab3dec65b8470d33cdd247b6740456891f70"}
Sep 09 12:58:08.119407 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:08.119120    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"
Sep 09 12:58:08.142956 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:08.142904    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6" podStartSLOduration=1.934945247 podStartE2EDuration="4.142886885s" totalImagesPullingTime="2.207941638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:58:08.142128829 +0000 UTC m=+571.062020168" watchObservedRunningTime="2026-09-09 12:58:08.142886885 +0000 UTC m=+571.062778223"
Sep 09 12:58:19.124055 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:19.124023    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"
Sep 09 12:58:37.506268 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:37.506240    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 12:58:37.509030 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:58:37.509007    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 12:59:22.466494 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:59:22.466459    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 12:59:55.859133 ip-10-0-140-133 kubenswrapper[2580]: I0909 12:59:55.859097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:00:00.131688 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:00.131648    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h"]
Sep 09 13:00:00.135105 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:00.135083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h"
Sep 09 13:00:00.137489 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:00.137468    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t5rn5\""
Sep 09 13:00:00.141143 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:00.141109    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h"]
Sep 09 13:00:00.147334 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:00.147313    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h"
Sep 09 13:00:00.419765 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:00.419698    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h"]
Sep 09 13:00:00.423327 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:00:00.423295    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9dc355_7e54_48f0_9e43_dceaa36ba16f.slice/crio-70fa8f5fff2b3f04f216116c169213bf15c6ee4a57840610ff68c209cf591bd8 WatchSource:0}: Error finding container 70fa8f5fff2b3f04f216116c169213bf15c6ee4a57840610ff68c209cf591bd8: Status 404 returned error can't find the container with id 70fa8f5fff2b3f04f216116c169213bf15c6ee4a57840610ff68c209cf591bd8
Sep 09 13:00:00.425331 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:00.425313    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 13:00:00.630398 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:00.630361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h" event={"ID":"ba9dc355-7e54-48f0-9e43-dceaa36ba16f","Type":"ContainerStarted","Data":"70fa8f5fff2b3f04f216116c169213bf15c6ee4a57840610ff68c209cf591bd8"}
Sep 09 13:00:03.647003 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:03.646914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h" event={"ID":"ba9dc355-7e54-48f0-9e43-dceaa36ba16f","Type":"ContainerStarted","Data":"a0c0de344db55e75c78b9ab7130866997b15863601a58d423901bc0e528c8c9a"}
Sep 09 13:00:03.660989 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:03.660940    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h" podStartSLOduration=0.771053014 podStartE2EDuration="3.660925976s" totalImagesPullingTime="2.889872962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:00:03.660389326 +0000 UTC m=+686.580280665" watchObservedRunningTime="2026-09-09 13:00:03.660925976 +0000 UTC m=+686.580817316"
Sep 09 13:00:04.654180 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:04.654141    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba9dc355-7e54-48f0-9e43-dceaa36ba16f" containerID="a0c0de344db55e75c78b9ab7130866997b15863601a58d423901bc0e528c8c9a" exitCode=0
Sep 09 13:00:04.654581 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:04.654241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h" event={"ID":"ba9dc355-7e54-48f0-9e43-dceaa36ba16f","Type":"ContainerDied","Data":"a0c0de344db55e75c78b9ab7130866997b15863601a58d423901bc0e528c8c9a"}
Sep 09 13:00:05.781789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:05.781764    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h"
Sep 09 13:00:06.664258 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.664226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h" event={"ID":"ba9dc355-7e54-48f0-9e43-dceaa36ba16f","Type":"ContainerDied","Data":"70fa8f5fff2b3f04f216116c169213bf15c6ee4a57840610ff68c209cf591bd8"}
Sep 09 13:00:06.664258 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.664259    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815980-glh9h"
Sep 09 13:00:06.664451 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.664271    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70fa8f5fff2b3f04f216116c169213bf15c6ee4a57840610ff68c209cf591bd8"
Sep 09 13:00:06.863063 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.863032    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"]
Sep 09 13:00:06.863657 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.863631    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba9dc355-7e54-48f0-9e43-dceaa36ba16f" containerName="cleanup"
Sep 09 13:00:06.863657 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.863656    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba9dc355-7e54-48f0-9e43-dceaa36ba16f" containerName="cleanup"
Sep 09 13:00:06.863846 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.863749    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba9dc355-7e54-48f0-9e43-dceaa36ba16f" containerName="cleanup"
Sep 09 13:00:06.872541 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.872518    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:06.875715 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.875693    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 13:00:06.876059 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.876036    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pjccl\""
Sep 09 13:00:06.876179 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.876118    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 13:00:06.876245 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.876041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 09 13:00:06.876734 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.876717    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"]
Sep 09 13:00:06.959053 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.958976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb87c7cd-e260-4632-ab78-cea8902c6e7f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:06.959053 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.959011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktx9c\" (UniqueName: \"kubernetes.io/projected/bb87c7cd-e260-4632-ab78-cea8902c6e7f-kube-api-access-ktx9c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:06.959053 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.959031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:06.959247 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.959055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:06.959247 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.959127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:06.959247 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:06.959154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.059861 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.059811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb87c7cd-e260-4632-ab78-cea8902c6e7f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.059963 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.059866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktx9c\" (UniqueName: \"kubernetes.io/projected/bb87c7cd-e260-4632-ab78-cea8902c6e7f-kube-api-access-ktx9c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.059963 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.059894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.059963 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.059910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.059963 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.059955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.060143 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.059982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.060417 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.060392    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.060417 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.060407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.060563 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.060427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.062254 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.062227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb87c7cd-e260-4632-ab78-cea8902c6e7f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.062494 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.062479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb87c7cd-e260-4632-ab78-cea8902c6e7f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.067668 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.067647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktx9c\" (UniqueName: \"kubernetes.io/projected/bb87c7cd-e260-4632-ab78-cea8902c6e7f-kube-api-access-ktx9c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npr5j\" (UID: \"bb87c7cd-e260-4632-ab78-cea8902c6e7f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.186386 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.186343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:07.458623 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.458599    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"]
Sep 09 13:00:07.460551 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:00:07.460528    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb87c7cd_e260_4632_ab78_cea8902c6e7f.slice/crio-75e305eae77e8762748999a3415f00f4ee0d31663447f6502cddce10b0d8f9e7 WatchSource:0}: Error finding container 75e305eae77e8762748999a3415f00f4ee0d31663447f6502cddce10b0d8f9e7: Status 404 returned error can't find the container with id 75e305eae77e8762748999a3415f00f4ee0d31663447f6502cddce10b0d8f9e7
Sep 09 13:00:07.567019 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.566986    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:00:07.669666 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:07.669635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j" event={"ID":"bb87c7cd-e260-4632-ab78-cea8902c6e7f","Type":"ContainerStarted","Data":"75e305eae77e8762748999a3415f00f4ee0d31663447f6502cddce10b0d8f9e7"}
Sep 09 13:00:11.692961 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:11.692860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j" event={"ID":"bb87c7cd-e260-4632-ab78-cea8902c6e7f","Type":"ContainerStarted","Data":"1c118bb88d79ef1d99c93c6ea853a3de3c3424324a65fe18036911e90c765831"}
Sep 09 13:00:11.710158 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:11.710097    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j" podStartSLOduration=1.863933759 podStartE2EDuration="5.710082012s" totalImagesPullingTime="3.846148253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:00:11.709288875 +0000 UTC m=+694.629180215" watchObservedRunningTime="2026-09-09 13:00:11.710082012 +0000 UTC m=+694.629973352"
Sep 09 13:00:17.186890 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:17.186854    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:17.187260 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:17.186941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:17.199342 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:17.199317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:17.736345 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:17.736314    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npr5j"
Sep 09 13:00:21.660464 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:21.660424    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:00:29.364486 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.364408    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"]
Sep 09 13:00:29.369901 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.369874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.372493 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.372469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 13:00:29.384471 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.384451    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"]
Sep 09 13:00:29.460339 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.460311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.460454 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.460378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.460454 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.460417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.460454 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.460442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnlvn\" (UniqueName: \"kubernetes.io/projected/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-kube-api-access-fnlvn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.460573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.460530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.460573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.460553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.560895 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.560871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.561008 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.560905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.561008 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.560928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnlvn\" (UniqueName: \"kubernetes.io/projected/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-kube-api-access-fnlvn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.561115 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.561008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.561115 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.561030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.561115 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.561068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.561307 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.561285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.561374 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.561305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.561432 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.561398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.563456 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.563436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.563602 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.563585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.569403 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.569381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnlvn\" (UniqueName: \"kubernetes.io/projected/4f26948a-bc1f-4e36-8a1a-9eb745e1f06a-kube-api-access-fnlvn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tgk4\" (UID: \"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.681805 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.681734    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:29.950778 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:29.950752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"]
Sep 09 13:00:29.953019 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:00:29.952992    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f26948a_bc1f_4e36_8a1a_9eb745e1f06a.slice/crio-700b30f07c3e9336200c8ddd873767f0fbdf88ed9162419d4877a1c494bf8832 WatchSource:0}: Error finding container 700b30f07c3e9336200c8ddd873767f0fbdf88ed9162419d4877a1c494bf8832: Status 404 returned error can't find the container with id 700b30f07c3e9336200c8ddd873767f0fbdf88ed9162419d4877a1c494bf8832
Sep 09 13:00:30.571242 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:30.571214    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:00:30.781899 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:30.781802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4" event={"ID":"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a","Type":"ContainerStarted","Data":"47aa722ebb202c7b15429ec01461518aaae40ebebbbaa3c060fcbc1f20393e64"}
Sep 09 13:00:30.781899 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:30.781886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4" event={"ID":"4f26948a-bc1f-4e36-8a1a-9eb745e1f06a","Type":"ContainerStarted","Data":"700b30f07c3e9336200c8ddd873767f0fbdf88ed9162419d4877a1c494bf8832"}
Sep 09 13:00:30.800921 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:30.800870    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4" podStartSLOduration=1.286101867 podStartE2EDuration="1.80085445s" totalImagesPullingTime="514.752583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:00:30.798580257 +0000 UTC m=+713.718471597" watchObservedRunningTime="2026-09-09 13:00:30.80085445 +0000 UTC m=+713.720745791"
Sep 09 13:00:38.566976 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:38.566939    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:00:39.681998 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:39.681966    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:39.682376 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:39.682034    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:39.694614 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:39.694588    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:39.833550 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:39.833516    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tgk4"
Sep 09 13:00:42.065278 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.065244    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"]
Sep 09 13:00:42.069835 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.069802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.072300 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.072274    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 13:00:42.078758 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.078731    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"]
Sep 09 13:00:42.177262 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.177223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hht6j\" (UniqueName: \"kubernetes.io/projected/78fbf9d4-ad4d-4f64-94df-da1fef0da984-kube-api-access-hht6j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.177467 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.177306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.177467 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.177385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.177467 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.177409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fbf9d4-ad4d-4f64-94df-da1fef0da984-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.177467 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.177440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.177622 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.177549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.278678 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.278631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.278678 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.278678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hht6j\" (UniqueName: \"kubernetes.io/projected/78fbf9d4-ad4d-4f64-94df-da1fef0da984-kube-api-access-hht6j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.278979 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.278718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.278979 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.278782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.278979 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.278808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fbf9d4-ad4d-4f64-94df-da1fef0da984-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.278979 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.278868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.279184 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.279137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.279184 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.279155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.279279 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.279237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.281376 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.281347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fbf9d4-ad4d-4f64-94df-da1fef0da984-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.281469 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.281436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fbf9d4-ad4d-4f64-94df-da1fef0da984-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.286770 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.286746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hht6j\" (UniqueName: \"kubernetes.io/projected/78fbf9d4-ad4d-4f64-94df-da1fef0da984-kube-api-access-hht6j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mqsqp\" (UID: \"78fbf9d4-ad4d-4f64-94df-da1fef0da984\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.381176 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.381078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:42.652500 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.652419    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"]
Sep 09 13:00:42.655466 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:00:42.655437    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78fbf9d4_ad4d_4f64_94df_da1fef0da984.slice/crio-43f0811cf6be7b23584c19c1fb48658949296e59c1b28933103ba31136a98755 WatchSource:0}: Error finding container 43f0811cf6be7b23584c19c1fb48658949296e59c1b28933103ba31136a98755: Status 404 returned error can't find the container with id 43f0811cf6be7b23584c19c1fb48658949296e59c1b28933103ba31136a98755
Sep 09 13:00:42.836929 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:42.836894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp" event={"ID":"78fbf9d4-ad4d-4f64-94df-da1fef0da984","Type":"ContainerStarted","Data":"43f0811cf6be7b23584c19c1fb48658949296e59c1b28933103ba31136a98755"}
Sep 09 13:00:43.843254 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:43.843219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp" event={"ID":"78fbf9d4-ad4d-4f64-94df-da1fef0da984","Type":"ContainerStarted","Data":"3017d86113c072c42e2c5e2242b07b8373140a11f16de22b17493c3460524c55"}
Sep 09 13:00:43.861176 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:43.861120    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp" podStartSLOduration=1.597500262 podStartE2EDuration="1.861101614s" totalImagesPullingTime="263.601352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:00:43.859218797 +0000 UTC m=+726.779110137" watchObservedRunningTime="2026-09-09 13:00:43.861101614 +0000 UTC m=+726.780992945"
Sep 09 13:00:44.076147 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:44.076095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:00:52.381257 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:52.381213    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:52.381710 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:52.381292    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:52.393908 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:52.393875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:52.898479 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:52.898446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mqsqp"
Sep 09 13:00:59.265185 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.265153    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"]
Sep 09 13:00:59.267948 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.267922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.270578 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.270555    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 13:00:59.277144 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.277118    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"]
Sep 09 13:00:59.440757 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.440716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.440757 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.440754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dzj\" (UniqueName: \"kubernetes.io/projected/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-kube-api-access-89dzj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.441025 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.440900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.441025 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.440934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.441025 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.440963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.441025 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.441011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.541977 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.541874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.541977 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.541921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dzj\" (UniqueName: \"kubernetes.io/projected/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-kube-api-access-89dzj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.542216 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.542033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.542216 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.542066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.542216 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.542091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.542216 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.542124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.542424 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.542276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.542488 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.542438    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.542543 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.542486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.544650 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.544627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.544742 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.544657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.551716 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.551696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dzj\" (UniqueName: \"kubernetes.io/projected/f3c58d0c-938f-4a0c-bb92-33a77cc8e677-kube-api-access-89dzj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n\" (UID: \"f3c58d0c-938f-4a0c-bb92-33a77cc8e677\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.578551 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.578522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:00:59.849365 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.849327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"]
Sep 09 13:00:59.852236 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:00:59.852204    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3c58d0c_938f_4a0c_bb92_33a77cc8e677.slice/crio-d61fdff2473d50b48b819836686efea7a66aa8c31043b8971f187d720427a3d1 WatchSource:0}: Error finding container d61fdff2473d50b48b819836686efea7a66aa8c31043b8971f187d720427a3d1: Status 404 returned error can't find the container with id d61fdff2473d50b48b819836686efea7a66aa8c31043b8971f187d720427a3d1
Sep 09 13:00:59.920673 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:00:59.920639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n" event={"ID":"f3c58d0c-938f-4a0c-bb92-33a77cc8e677","Type":"ContainerStarted","Data":"d61fdff2473d50b48b819836686efea7a66aa8c31043b8971f187d720427a3d1"}
Sep 09 13:01:00.927684 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:01:00.927645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n" event={"ID":"f3c58d0c-938f-4a0c-bb92-33a77cc8e677","Type":"ContainerStarted","Data":"42e23b415bcb0fbfac35483d8351d8d24a50593f11c6ed7f034ab38bb7866dc1"}
Sep 09 13:01:00.944957 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:01:00.944903    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n" podStartSLOduration=1.640044016 podStartE2EDuration="1.94488757s" totalImagesPullingTime="304.843554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:01:00.944361731 +0000 UTC m=+743.864253070" watchObservedRunningTime="2026-09-09 13:01:00.94488757 +0000 UTC m=+743.864778906"
Sep 09 13:01:01.468230 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:01:01.468190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:01:09.579326 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:01:09.579274    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:01:09.579735 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:01:09.579546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:01:09.593033 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:01:09.593005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:01:09.980173 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:01:09.980089    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n"
Sep 09 13:03:09.186053 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:09.186018    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:03:17.778128 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.778096    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"]
Sep 09 13:03:17.781293 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.781272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.783903 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.783880    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d0fb1d64-serving-cert\""
Sep 09 13:03:17.784157 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.784143    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 13:03:17.785081 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.785022    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:03:17.785203 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.785114    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dbjk4\""
Sep 09 13:03:17.785203 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.785132    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 13:03:17.790241 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.789997    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"]
Sep 09 13:03:17.844706 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.844678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-maas-api-tls\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.844845 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.844716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-metrics-certs\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.844845 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.844771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjd9\" (UniqueName: \"kubernetes.io/projected/4a0d9cc2-9a1e-4354-92a5-6783e67add27-kube-api-access-lnjd9\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.844945 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.844857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0d9cc2-9a1e-4354-92a5-6783e67add27-openshift-service-ca\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.946158 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.946134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-maas-api-tls\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.946270 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.946163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-metrics-certs\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.946270 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.946198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjd9\" (UniqueName: \"kubernetes.io/projected/4a0d9cc2-9a1e-4354-92a5-6783e67add27-kube-api-access-lnjd9\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.946462 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.946441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0d9cc2-9a1e-4354-92a5-6783e67add27-openshift-service-ca\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.947091 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.947018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0d9cc2-9a1e-4354-92a5-6783e67add27-openshift-service-ca\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.948573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.948552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-metrics-certs\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.948839 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.948803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-maas-api-tls\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:17.954121 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:17.954098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjd9\" (UniqueName: \"kubernetes.io/projected/4a0d9cc2-9a1e-4354-92a5-6783e67add27-kube-api-access-lnjd9\") pod \"maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:18.095559 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:18.095526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:18.366248 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:18.366222    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"]
Sep 09 13:03:18.369035 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:03:18.369002    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0d9cc2_9a1e_4354_92a5_6783e67add27.slice/crio-1cc0620b286dd879d5b93ce4efb9c18257a3b9e4da7bb6611a57aa3ae0ddd42c WatchSource:0}: Error finding container 1cc0620b286dd879d5b93ce4efb9c18257a3b9e4da7bb6611a57aa3ae0ddd42c: Status 404 returned error can't find the container with id 1cc0620b286dd879d5b93ce4efb9c18257a3b9e4da7bb6611a57aa3ae0ddd42c
Sep 09 13:03:18.582911 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:18.582876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc" event={"ID":"4a0d9cc2-9a1e-4354-92a5-6783e67add27","Type":"ContainerStarted","Data":"1cc0620b286dd879d5b93ce4efb9c18257a3b9e4da7bb6611a57aa3ae0ddd42c"}
Sep 09 13:03:21.601915 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:21.601876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc" event={"ID":"4a0d9cc2-9a1e-4354-92a5-6783e67add27","Type":"ContainerStarted","Data":"d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210"}
Sep 09 13:03:21.602398 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:21.602033    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:21.618720 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:21.618631    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc" podStartSLOduration=2.374077433 podStartE2EDuration="4.618614587s" totalImagesPullingTime="2.244537154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:03:21.617657376 +0000 UTC m=+884.537548718" watchObservedRunningTime="2026-09-09 13:03:21.618614587 +0000 UTC m=+884.538505925"
Sep 09 13:03:22.765093 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:22.765060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:03:27.612066 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:27.612037    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:34.705215 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.705145    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"]
Sep 09 13:03:34.708241 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.708221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.711312 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.711291    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 13:03:34.711553 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.711538    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7c991b-openshift-default-dockercfg-8mpdm\""
Sep 09 13:03:34.716949 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.716927    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"]
Sep 09 13:03:34.770467 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.770438    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:03:34.897906 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.897881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b2dea0-ebbb-4278-accb-c21559060609-istiod-ca-cert\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.898015 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.897927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lbts\" (UniqueName: \"kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-kube-api-access-9lbts\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.898015 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.897967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b2dea0-ebbb-4278-accb-c21559060609-istio-podinfo\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.898083 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.898062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-credential-socket\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.898120 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.898090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-istio-token\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.898120 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.898110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-socket\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.898275 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.898262    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-envoy\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.898310 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.898291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-data\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.898346 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.898307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-certs\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.999679 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.999627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b2dea0-ebbb-4278-accb-c21559060609-istiod-ca-cert\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.999679 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.999669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lbts\" (UniqueName: \"kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-kube-api-access-9lbts\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.999809 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.999715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b2dea0-ebbb-4278-accb-c21559060609-istio-podinfo\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.999809 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.999754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-credential-socket\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.999809 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.999783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-istio-token\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.999992 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.999810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-socket\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.999992 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.999896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-envoy\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.999992 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.999942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-data\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:34.999992 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:34.999969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-certs\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:35.000179 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.000126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-credential-socket\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:35.000330 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.000305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-socket\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:35.000375 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.000325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-certs\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:35.000417 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.000368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-data\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:35.000457 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.000431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b2dea0-ebbb-4278-accb-c21559060609-istiod-ca-cert\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:35.002238 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.002214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-envoy\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:35.002492 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.002473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b2dea0-ebbb-4278-accb-c21559060609-istio-podinfo\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:35.008025 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.007998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-istio-token\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:35.008709 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.008687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lbts\" (UniqueName: \"kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-kube-api-access-9lbts\") pod \"e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") " pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:35.037632 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.037610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:35.310237 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.310216    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"]
Sep 09 13:03:35.312660 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:03:35.312630    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b2dea0_ebbb_4278_accb_c21559060609.slice/crio-b8c5cb4ca7d73110378a96cfe929e04e448821b2c44c74ff766ab5d8d1b74cb3 WatchSource:0}: Error finding container b8c5cb4ca7d73110378a96cfe929e04e448821b2c44c74ff766ab5d8d1b74cb3: Status 404 returned error can't find the container with id b8c5cb4ca7d73110378a96cfe929e04e448821b2c44c74ff766ab5d8d1b74cb3
Sep 09 13:03:35.670174 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:35.670092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" event={"ID":"02b2dea0-ebbb-4278-accb-c21559060609","Type":"ContainerStarted","Data":"b8c5cb4ca7d73110378a96cfe929e04e448821b2c44c74ff766ab5d8d1b74cb3"}
Sep 09 13:03:38.111632 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:38.111599    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 13:03:38.112127 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:38.112103    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 13:03:38.127592 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:38.127558    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:03:38.127682 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:38.127628    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:03:38.127682 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:38.127654    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:03:38.687382 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:38.687351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" event={"ID":"02b2dea0-ebbb-4278-accb-c21559060609","Type":"ContainerStarted","Data":"85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2"}
Sep 09 13:03:38.709760 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:38.709702    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" podStartSLOduration=1.897111499 podStartE2EDuration="4.709683297s" totalImagesPullingTime="2.812571798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:03:38.707649254 +0000 UTC m=+901.627540605" watchObservedRunningTime="2026-09-09 13:03:38.709683297 +0000 UTC m=+901.629574639"
Sep 09 13:03:39.037916 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:39.037814    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:39.042188 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:39.042161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:39.696076 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:39.696043    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:39.697206 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:39.697174    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:03:42.078583 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.078551    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"]
Sep 09 13:03:42.081904 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.081888    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"
Sep 09 13:03:42.084328 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.084308    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t5rn5\""
Sep 09 13:03:42.089725 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.089704    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"]
Sep 09 13:03:42.163712 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.163680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6f0039-957a-4747-8852-05284a2e4941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d\" (UID: \"3a6f0039-957a-4747-8852-05284a2e4941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"
Sep 09 13:03:42.264628 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.264593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6f0039-957a-4747-8852-05284a2e4941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d\" (UID: \"3a6f0039-957a-4747-8852-05284a2e4941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"
Sep 09 13:03:42.265233 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.265210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6f0039-957a-4747-8852-05284a2e4941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d\" (UID: \"3a6f0039-957a-4747-8852-05284a2e4941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"
Sep 09 13:03:42.394409 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.394328    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"
Sep 09 13:03:42.635422 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.635394    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"]
Sep 09 13:03:42.638428 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.638411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:03:42.642381 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.642359    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7c991b-dockercfg-lj7hr\""
Sep 09 13:03:42.642555 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.642539    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7c991b\""
Sep 09 13:03:42.650163 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.650140    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"]
Sep 09 13:03:42.668002 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.667972    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"]
Sep 09 13:03:42.668455 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.668429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80bba08-fd5a-4070-9511-aab8692e0681-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth\" (UID: \"a80bba08-fd5a-4070-9511-aab8692e0681\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:03:42.668872 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.668847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48h4d\" (UniqueName: \"kubernetes.io/projected/a80bba08-fd5a-4070-9511-aab8692e0681-kube-api-access-48h4d\") pod \"payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth\" (UID: \"a80bba08-fd5a-4070-9511-aab8692e0681\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:03:42.671716 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:03:42.671695    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6f0039_957a_4747_8852_05284a2e4941.slice/crio-e9bb25b2764694a170428dbe888e605d18e08013e9bd3b54551a2ddb52176c4c WatchSource:0}: Error finding container e9bb25b2764694a170428dbe888e605d18e08013e9bd3b54551a2ddb52176c4c: Status 404 returned error can't find the container with id e9bb25b2764694a170428dbe888e605d18e08013e9bd3b54551a2ddb52176c4c
Sep 09 13:03:42.710356 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.710316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d" event={"ID":"3a6f0039-957a-4747-8852-05284a2e4941","Type":"ContainerStarted","Data":"e9bb25b2764694a170428dbe888e605d18e08013e9bd3b54551a2ddb52176c4c"}
Sep 09 13:03:42.770060 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.770020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48h4d\" (UniqueName: \"kubernetes.io/projected/a80bba08-fd5a-4070-9511-aab8692e0681-kube-api-access-48h4d\") pod \"payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth\" (UID: \"a80bba08-fd5a-4070-9511-aab8692e0681\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:03:42.770225 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.770128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80bba08-fd5a-4070-9511-aab8692e0681-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth\" (UID: \"a80bba08-fd5a-4070-9511-aab8692e0681\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:03:42.770810 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.770790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80bba08-fd5a-4070-9511-aab8692e0681-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth\" (UID: \"a80bba08-fd5a-4070-9511-aab8692e0681\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:03:42.778993 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.778961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48h4d\" (UniqueName: \"kubernetes.io/projected/a80bba08-fd5a-4070-9511-aab8692e0681-kube-api-access-48h4d\") pod \"payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth\" (UID: \"a80bba08-fd5a-4070-9511-aab8692e0681\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:03:42.950767 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:42.950736    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:03:43.223154 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:43.223127    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"]
Sep 09 13:03:43.225662 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:03:43.225637    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80bba08_fd5a_4070_9511_aab8692e0681.slice/crio-912b3e12e3a0e6c44945c854410fb12a97720864d9222ab0c013739f78969971 WatchSource:0}: Error finding container 912b3e12e3a0e6c44945c854410fb12a97720864d9222ab0c013739f78969971: Status 404 returned error can't find the container with id 912b3e12e3a0e6c44945c854410fb12a97720864d9222ab0c013739f78969971
Sep 09 13:03:43.717583 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:43.717554    2580 generic.go:356] "Generic (PLEG): container finished" podID="3a6f0039-957a-4747-8852-05284a2e4941" containerID="e1ca47866070096f444dc72b5827183f85cfa513bfa014e6df0415516967b34c" exitCode=0
Sep 09 13:03:43.717747 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:43.717625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d" event={"ID":"3a6f0039-957a-4747-8852-05284a2e4941","Type":"ContainerDied","Data":"e1ca47866070096f444dc72b5827183f85cfa513bfa014e6df0415516967b34c"}
Sep 09 13:03:43.718777 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:43.718746    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth" event={"ID":"a80bba08-fd5a-4070-9511-aab8692e0681","Type":"ContainerStarted","Data":"912b3e12e3a0e6c44945c854410fb12a97720864d9222ab0c013739f78969971"}
Sep 09 13:03:45.851426 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:45.851402    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"
Sep 09 13:03:45.900638 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:45.900614    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6f0039-957a-4747-8852-05284a2e4941-maas-api-service-ca\") pod \"3a6f0039-957a-4747-8852-05284a2e4941\" (UID: \"3a6f0039-957a-4747-8852-05284a2e4941\") "
Sep 09 13:03:45.900978 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:45.900957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6f0039-957a-4747-8852-05284a2e4941-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a6f0039-957a-4747-8852-05284a2e4941" (UID: "3a6f0039-957a-4747-8852-05284a2e4941"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:03:46.001904 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:46.001873    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6f0039-957a-4747-8852-05284a2e4941-maas-api-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:03:46.739884 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:46.739851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d" event={"ID":"3a6f0039-957a-4747-8852-05284a2e4941","Type":"ContainerDied","Data":"e9bb25b2764694a170428dbe888e605d18e08013e9bd3b54551a2ddb52176c4c"}
Sep 09 13:03:46.740094 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:46.739915    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9bb25b2764694a170428dbe888e605d18e08013e9bd3b54551a2ddb52176c4c"
Sep 09 13:03:46.740094 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:46.739866    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"
Sep 09 13:03:46.741519 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:46.741484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth" event={"ID":"a80bba08-fd5a-4070-9511-aab8692e0681","Type":"ContainerStarted","Data":"db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e"}
Sep 09 13:03:46.741645 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:46.741629    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:03:46.759869 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:46.759808    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth" podStartSLOduration=2.122686044 podStartE2EDuration="4.759795126s" totalImagesPullingTime="2.637109082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:03:46.757666317 +0000 UTC m=+909.677557656" watchObservedRunningTime="2026-09-09 13:03:46.759795126 +0000 UTC m=+909.679686465"
Sep 09 13:03:48.402524 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.402490    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"]
Sep 09 13:03:48.402987 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.402699    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc" podUID="4a0d9cc2-9a1e-4354-92a5-6783e67add27" containerName="maas-api" containerID="cri-o://d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210" gracePeriod=30
Sep 09 13:03:48.641541 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.641517    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:48.729378 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.729299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-metrics-certs\") pod \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") "
Sep 09 13:03:48.729526 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.729418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-maas-api-tls\") pod \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") "
Sep 09 13:03:48.729585 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.729541    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0d9cc2-9a1e-4354-92a5-6783e67add27-openshift-service-ca\") pod \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") "
Sep 09 13:03:48.729585 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.729569    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnjd9\" (UniqueName: \"kubernetes.io/projected/4a0d9cc2-9a1e-4354-92a5-6783e67add27-kube-api-access-lnjd9\") pod \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\" (UID: \"4a0d9cc2-9a1e-4354-92a5-6783e67add27\") "
Sep 09 13:03:48.729905 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.729875    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0d9cc2-9a1e-4354-92a5-6783e67add27-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a0d9cc2-9a1e-4354-92a5-6783e67add27" (UID: "4a0d9cc2-9a1e-4354-92a5-6783e67add27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:03:48.731623 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.731596    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a0d9cc2-9a1e-4354-92a5-6783e67add27" (UID: "4a0d9cc2-9a1e-4354-92a5-6783e67add27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:03:48.731712 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.731621    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4a0d9cc2-9a1e-4354-92a5-6783e67add27" (UID: "4a0d9cc2-9a1e-4354-92a5-6783e67add27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:03:48.731755 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.731727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0d9cc2-9a1e-4354-92a5-6783e67add27-kube-api-access-lnjd9" (OuterVolumeSpecName: "kube-api-access-lnjd9") pod "4a0d9cc2-9a1e-4354-92a5-6783e67add27" (UID: "4a0d9cc2-9a1e-4354-92a5-6783e67add27"). InnerVolumeSpecName "kube-api-access-lnjd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:03:48.752476 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.752444    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a0d9cc2-9a1e-4354-92a5-6783e67add27" containerID="d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210" exitCode=0
Sep 09 13:03:48.752561 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.752490    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"
Sep 09 13:03:48.752561 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.752518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc" event={"ID":"4a0d9cc2-9a1e-4354-92a5-6783e67add27","Type":"ContainerDied","Data":"d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210"}
Sep 09 13:03:48.752625 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.752571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc" event={"ID":"4a0d9cc2-9a1e-4354-92a5-6783e67add27","Type":"ContainerDied","Data":"1cc0620b286dd879d5b93ce4efb9c18257a3b9e4da7bb6611a57aa3ae0ddd42c"}
Sep 09 13:03:48.752625 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.752593    2580 scope.go:117] "RemoveContainer" containerID="d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210"
Sep 09 13:03:48.761572 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.761557    2580 scope.go:117] "RemoveContainer" containerID="d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210"
Sep 09 13:03:48.761849 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:03:48.761812    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210\": container with ID starting with d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210 not found: ID does not exist" containerID="d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210"
Sep 09 13:03:48.761913 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.761857    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210"} err="failed to get container status \"d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210\": rpc error: code = NotFound desc = could not find container \"d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210\": container with ID starting with d8969820f12cee5bac580e2bd811a0601a22e76fdc9fe1a0b545f59a13425210 not found: ID does not exist"
Sep 09 13:03:48.773603 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.773583    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"]
Sep 09 13:03:48.778135 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.778113    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"]
Sep 09 13:03:48.786601 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.783777    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0fb1d64-86c57649c6-r6fcc"]
Sep 09 13:03:48.789766 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.789747    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0fb1d64-xjw2d"]
Sep 09 13:03:48.830465 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.830438    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0d9cc2-9a1e-4354-92a5-6783e67add27-openshift-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:03:48.830465 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.830462    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnjd9\" (UniqueName: \"kubernetes.io/projected/4a0d9cc2-9a1e-4354-92a5-6783e67add27-kube-api-access-lnjd9\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:03:48.830585 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.830479    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-metrics-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:03:48.830585 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:48.830491    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0d9cc2-9a1e-4354-92a5-6783e67add27-maas-api-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:03:49.591678 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:49.591644    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a6f0039-957a-4747-8852-05284a2e4941" path="/var/lib/kubelet/pods/3a6f0039-957a-4747-8852-05284a2e4941/volumes"
Sep 09 13:03:49.592092 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:49.592078    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a0d9cc2-9a1e-4354-92a5-6783e67add27" path="/var/lib/kubelet/pods/4a0d9cc2-9a1e-4354-92a5-6783e67add27/volumes"
Sep 09 13:03:49.961416 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:49.961328    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"]
Sep 09 13:03:49.962022 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:49.961999    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a0d9cc2-9a1e-4354-92a5-6783e67add27" containerName="maas-api"
Sep 09 13:03:49.962022 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:49.962022    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a0d9cc2-9a1e-4354-92a5-6783e67add27" containerName="maas-api"
Sep 09 13:03:49.962199 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:49.962038    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6f0039-957a-4747-8852-05284a2e4941" containerName="revoke-keys"
Sep 09 13:03:49.962199 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:49.962048    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6f0039-957a-4747-8852-05284a2e4941" containerName="revoke-keys"
Sep 09 13:03:49.962199 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:49.962185    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a0d9cc2-9a1e-4354-92a5-6783e67add27" containerName="maas-api"
Sep 09 13:03:49.962199 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:49.962196    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6f0039-957a-4747-8852-05284a2e4941" containerName="revoke-keys"
Sep 09 13:03:49.965757 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:49.965733    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:49.968696 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:49.968678    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-776d31f2-gw-openshift-default-dockercfg-5q854\""
Sep 09 13:03:50.014101 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.014079    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"]
Sep 09 13:03:50.041047 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.041023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-podinfo\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.041150 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.041063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-data\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.041150 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.041094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-envoy\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.041228 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.041177    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-token\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.041228 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.041215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-socket\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.041300 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.041244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-credential-socket\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.041300 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.041264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af351b4-6747-4692-83a0-bbb2db0a1a63-istiod-ca-cert\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.041406 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.041347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-certs\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.041406 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.041386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhcp6\" (UniqueName: \"kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-kube-api-access-bhcp6\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.142471 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.142438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-envoy\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.142602 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.142500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-token\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.142602 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.142531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-socket\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.142602 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.142567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-credential-socket\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.142602 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.142598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af351b4-6747-4692-83a0-bbb2db0a1a63-istiod-ca-cert\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.142794 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.142640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-certs\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.142794 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.142667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhcp6\" (UniqueName: \"kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-kube-api-access-bhcp6\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.142794 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.142710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-podinfo\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.142794 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.142757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-data\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.143013 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.142984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-credential-socket\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.143071 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.143050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-socket\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.143316 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.143294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-certs\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.143390 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.143360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-data\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.143518 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.143404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af351b4-6747-4692-83a0-bbb2db0a1a63-istiod-ca-cert\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.145065 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.145042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-envoy\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.145431 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.145412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-podinfo\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.167893 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.167867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhcp6\" (UniqueName: \"kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-kube-api-access-bhcp6\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.167999 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.167986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-token\") pod \"e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") " pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.277875 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.277776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:50.557605 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.557578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"]
Sep 09 13:03:50.559274 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:03:50.559246    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af351b4_6747_4692_83a0_bbb2db0a1a63.slice/crio-46340d1065d42eba21b475864ebf4a56c9947f195b48d3fc94a5d9941024558f WatchSource:0}: Error finding container 46340d1065d42eba21b475864ebf4a56c9947f195b48d3fc94a5d9941024558f: Status 404 returned error can't find the container with id 46340d1065d42eba21b475864ebf4a56c9947f195b48d3fc94a5d9941024558f
Sep 09 13:03:50.561463 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.561433    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:03:50.561526 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.561517    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:03:50.561571 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.561544    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:03:50.765489 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.765454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" event={"ID":"5af351b4-6747-4692-83a0-bbb2db0a1a63","Type":"ContainerStarted","Data":"69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b"}
Sep 09 13:03:50.765843 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.765512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" event={"ID":"5af351b4-6747-4692-83a0-bbb2db0a1a63","Type":"ContainerStarted","Data":"46340d1065d42eba21b475864ebf4a56c9947f195b48d3fc94a5d9941024558f"}
Sep 09 13:03:50.786440 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.786383    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" podStartSLOduration=1.7863620770000002 podStartE2EDuration="1.786362077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:03:50.785039692 +0000 UTC m=+913.704931042" watchObservedRunningTime="2026-09-09 13:03:50.786362077 +0000 UTC m=+913.706253418"
Sep 09 13:03:50.890021 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:50.889947    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:03:51.278046 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:51.277953    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:51.283233 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:51.283212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:51.770273 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:51.770244    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:51.771239 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:51.771220    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:03:57.748752 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:57.748720    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:03:59.648496 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.648462    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"]
Sep 09 13:03:59.655542 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.655522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.658094 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.658074    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-a333cccb-kserve-self-signed-certs\""
Sep 09 13:03:59.661389 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.661368    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"]
Sep 09 13:03:59.745283 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.745240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-dshm\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.745463 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.745289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-model-cache\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.745463 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.745315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw45c\" (UniqueName: \"kubernetes.io/projected/7fdb3882-8344-4d47-a769-e7d20ee77a97-kube-api-access-lw45c\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.745463 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.745414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-tmp-dir\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.745594 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.745495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fdb3882-8344-4d47-a769-e7d20ee77a97-tls-certs\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.745594 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.745544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-home\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.846250 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.846216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-tmp-dir\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.846421 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.846306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fdb3882-8344-4d47-a769-e7d20ee77a97-tls-certs\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.846421 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.846371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-home\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.846421 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.846402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-dshm\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.846575 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.846434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-model-cache\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.846575 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.846460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw45c\" (UniqueName: \"kubernetes.io/projected/7fdb3882-8344-4d47-a769-e7d20ee77a97-kube-api-access-lw45c\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.846678 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.846643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-tmp-dir\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.846803 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.846694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-home\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.846904 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.846881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-model-cache\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.848789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.848769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-dshm\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.848979 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.848962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fdb3882-8344-4d47-a769-e7d20ee77a97-tls-certs\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.857287 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.857258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw45c\" (UniqueName: \"kubernetes.io/projected/7fdb3882-8344-4d47-a769-e7d20ee77a97-kube-api-access-lw45c\") pod \"e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") " pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:03:59.967723 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:03:59.967623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:04:00.250531 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:00.250503    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"]
Sep 09 13:04:00.252365 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:04:00.252341    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fdb3882_8344_4d47_a769_e7d20ee77a97.slice/crio-b88b93fbabf6df19fc27abe21501d02debbe202ab24ba75e53f708e7634fe341 WatchSource:0}: Error finding container b88b93fbabf6df19fc27abe21501d02debbe202ab24ba75e53f708e7634fe341: Status 404 returned error can't find the container with id b88b93fbabf6df19fc27abe21501d02debbe202ab24ba75e53f708e7634fe341
Sep 09 13:04:00.816516 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:00.816467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh" event={"ID":"7fdb3882-8344-4d47-a769-e7d20ee77a97","Type":"ContainerStarted","Data":"b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e"}
Sep 09 13:04:00.816913 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:00.816525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh" event={"ID":"7fdb3882-8344-4d47-a769-e7d20ee77a97","Type":"ContainerStarted","Data":"b88b93fbabf6df19fc27abe21501d02debbe202ab24ba75e53f708e7634fe341"}
Sep 09 13:04:00.832469 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:00.832423    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh" podStartSLOduration=1.832407997 podStartE2EDuration="1.832407997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:04:00.831555361 +0000 UTC m=+923.751446692" watchObservedRunningTime="2026-09-09 13:04:00.832407997 +0000 UTC m=+923.752299336"
Sep 09 13:04:01.822654 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:01.822623    2580 generic.go:356] "Generic (PLEG): container finished" podID="02b2dea0-ebbb-4278-accb-c21559060609" containerID="85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2" exitCode=0
Sep 09 13:04:01.823109 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:01.822698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" event={"ID":"02b2dea0-ebbb-4278-accb-c21559060609","Type":"ContainerDied","Data":"85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2"}
Sep 09 13:04:01.823212 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:01.823195    2580 scope.go:117] "RemoveContainer" containerID="85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2"
Sep 09 13:04:01.824318 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:01.824294    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:04:01.824381 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:01.824356    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:04:01.824420 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:01.824387    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:04:02.833784 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:02.833735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" event={"ID":"02b2dea0-ebbb-4278-accb-c21559060609","Type":"ContainerStarted","Data":"441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581"}
Sep 09 13:04:03.038894 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:03.038781    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:04:03.039126 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:03.039080    2580 patch_prober.go:28] interesting pod/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:04:03.039219 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:03.039162    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 09 13:04:04.038836 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:04.038770    2580 patch_prober.go:28] interesting pod/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:04:04.039334 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:04.038861    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 09 13:04:05.038419 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:05.038382    2580 patch_prober.go:28] interesting pod/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:04:05.038708 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:05.038682    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 09 13:04:06.039040 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:06.039002    2580 patch_prober.go:28] interesting pod/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:04:06.039625 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:06.039592    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 09 13:04:07.038912 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:07.038871    2580 patch_prober.go:28] interesting pod/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:04:07.039132 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:07.038940    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 09 13:04:07.471870 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:07.471827    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:04:08.038040 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:08.038003    2580 patch_prober.go:28] interesting pod/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:04:08.038218 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:08.038060    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 09 13:04:09.043250 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:09.043217    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:04:09.043629 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:09.043523    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:04:09.044350 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:09.044331    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:04:09.967946 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:09.967909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:04:09.967946 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:09.967956    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:04:09.980611 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:09.980590    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:04:10.891749 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:10.891717    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:04:11.886489 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:11.886459    2580 generic.go:356] "Generic (PLEG): container finished" podID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerID="69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b" exitCode=0
Sep 09 13:04:11.886680 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:11.886532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" event={"ID":"5af351b4-6747-4692-83a0-bbb2db0a1a63","Type":"ContainerDied","Data":"69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b"}
Sep 09 13:04:11.887006 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:11.886991    2580 scope.go:117] "RemoveContainer" containerID="69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b"
Sep 09 13:04:11.887976 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:11.887951    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:04:11.888045 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:11.888013    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:04:11.888079 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:11.888046    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:04:12.894758 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:12.894724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" event={"ID":"5af351b4-6747-4692-83a0-bbb2db0a1a63","Type":"ContainerStarted","Data":"2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f"}
Sep 09 13:04:13.278291 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:13.278201    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:04:13.278507 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:13.278484    2580 patch_prober.go:28] interesting pod/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 09 13:04:13.278561 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:13.278542    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 09 13:04:14.278401 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:14.278360    2580 patch_prober.go:28] interesting pod/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 09 13:04:14.278806 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:14.278430    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 09 13:04:15.278311 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:15.278275    2580 patch_prober.go:28] interesting pod/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 09 13:04:15.278489 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:15.278336    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 09 13:04:16.278803 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:16.278755    2580 patch_prober.go:28] interesting pod/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 09 13:04:16.279184 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:16.278839    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 09 13:04:17.278624 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.278587    2580 patch_prober.go:28] interesting pod/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 09 13:04:17.278847 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.278656    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 09 13:04:17.340463 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.340426    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"]
Sep 09 13:04:17.346671 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.346643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"
Sep 09 13:04:17.349451 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.349420    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"]
Sep 09 13:04:17.349774 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.349665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:04:17.349918 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.349899    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t5rn5\""
Sep 09 13:04:17.428690 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.428651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5add63-9c49-43d9-aca4-a80bc37d16f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-776d31f2-567gv\" (UID: \"7b5add63-9c49-43d9-aca4-a80bc37d16f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"
Sep 09 13:04:17.529490 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.529401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5add63-9c49-43d9-aca4-a80bc37d16f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-776d31f2-567gv\" (UID: \"7b5add63-9c49-43d9-aca4-a80bc37d16f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"
Sep 09 13:04:17.530067 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.530046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5add63-9c49-43d9-aca4-a80bc37d16f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-776d31f2-567gv\" (UID: \"7b5add63-9c49-43d9-aca4-a80bc37d16f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"
Sep 09 13:04:17.658293 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.658265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"
Sep 09 13:04:17.949549 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:17.949525    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"]
Sep 09 13:04:17.952553 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:04:17.952526    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5add63_9c49_43d9_aca4_a80bc37d16f4.slice/crio-1d10c4de15cf2dfe7be76e1662ee575e2d18438038e2e8bc10bda722d10b6d3a WatchSource:0}: Error finding container 1d10c4de15cf2dfe7be76e1662ee575e2d18438038e2e8bc10bda722d10b6d3a: Status 404 returned error can't find the container with id 1d10c4de15cf2dfe7be76e1662ee575e2d18438038e2e8bc10bda722d10b6d3a
Sep 09 13:04:18.278878 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:18.278775    2580 patch_prober.go:28] interesting pod/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 09 13:04:18.278878 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:18.278833    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 09 13:04:18.944280 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:18.944242    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b5add63-9c49-43d9-aca4-a80bc37d16f4" containerID="c3c6f881742a7c2769e6111110880a999f88638cc4bd48fb0d1b69edf67b824f" exitCode=0
Sep 09 13:04:18.944425 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:18.944331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv" event={"ID":"7b5add63-9c49-43d9-aca4-a80bc37d16f4","Type":"ContainerDied","Data":"c3c6f881742a7c2769e6111110880a999f88638cc4bd48fb0d1b69edf67b824f"}
Sep 09 13:04:18.944464 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:18.944421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv" event={"ID":"7b5add63-9c49-43d9-aca4-a80bc37d16f4","Type":"ContainerStarted","Data":"1d10c4de15cf2dfe7be76e1662ee575e2d18438038e2e8bc10bda722d10b6d3a"}
Sep 09 13:04:19.282583 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:19.282502    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:04:19.282988 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:19.282791    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:04:19.283543 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:19.283524    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:04:20.080955 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:20.080932    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"
Sep 09 13:04:20.154856 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:20.154808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5add63-9c49-43d9-aca4-a80bc37d16f4-maas-api-service-ca\") pod \"7b5add63-9c49-43d9-aca4-a80bc37d16f4\" (UID: \"7b5add63-9c49-43d9-aca4-a80bc37d16f4\") "
Sep 09 13:04:20.155117 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:20.155098    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5add63-9c49-43d9-aca4-a80bc37d16f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b5add63-9c49-43d9-aca4-a80bc37d16f4" (UID: "7b5add63-9c49-43d9-aca4-a80bc37d16f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:04:20.256344 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:20.256290    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5add63-9c49-43d9-aca4-a80bc37d16f4-maas-api-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:20.955350 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:20.955317    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"
Sep 09 13:04:20.955711 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:20.955359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv" event={"ID":"7b5add63-9c49-43d9-aca4-a80bc37d16f4","Type":"ContainerDied","Data":"1d10c4de15cf2dfe7be76e1662ee575e2d18438038e2e8bc10bda722d10b6d3a"}
Sep 09 13:04:20.955711 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:20.955400    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d10c4de15cf2dfe7be76e1662ee575e2d18438038e2e8bc10bda722d10b6d3a"
Sep 09 13:04:23.930799 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:23.930764    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"]
Sep 09 13:04:23.932730 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:23.932707    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-776d31f2-567gv"]
Sep 09 13:04:24.071375 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:24.071339    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"]
Sep 09 13:04:24.071604 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:24.071578    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy" containerID="cri-o://2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f" gracePeriod=30
Sep 09 13:04:25.591533 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:25.591500    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b5add63-9c49-43d9-aca4-a80bc37d16f4" path="/var/lib/kubelet/pods/7b5add63-9c49-43d9-aca4-a80bc37d16f4/volumes"
Sep 09 13:04:28.649340 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.649293    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"]
Sep 09 13:04:28.649940 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.649921    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5add63-9c49-43d9-aca4-a80bc37d16f4" containerName="revoke-keys"
Sep 09 13:04:28.650030 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.649943    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5add63-9c49-43d9-aca4-a80bc37d16f4" containerName="revoke-keys"
Sep 09 13:04:28.650091 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.650079    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5add63-9c49-43d9-aca4-a80bc37d16f4" containerName="revoke-keys"
Sep 09 13:04:28.652490 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.652468    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:28.655717 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.655696    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0175d9d3\""
Sep 09 13:04:28.664778 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.664750    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"]
Sep 09 13:04:28.701756 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.701712    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"]
Sep 09 13:04:28.704706 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.704689    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:28.709514 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.709492    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0175d9d3-dockercfg-cwcxb\""
Sep 09 13:04:28.718900 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.718873    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"]
Sep 09 13:04:28.738892 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.738849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60a7d133-03eb-4556-ba60-11a81566bdfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb\" (UID: \"60a7d133-03eb-4556-ba60-11a81566bdfc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:28.739068 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.738967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nrp8\" (UniqueName: \"kubernetes.io/projected/60a7d133-03eb-4556-ba60-11a81566bdfc-kube-api-access-4nrp8\") pod \"payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb\" (UID: \"60a7d133-03eb-4556-ba60-11a81566bdfc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:28.840376 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.840327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fc2359-7150-4753-b04a-e4f618f47fb3-plugins-config-volume\") pod \"payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn\" (UID: \"22fc2359-7150-4753-b04a-e4f618f47fb3\") " pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:28.840559 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.840411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60a7d133-03eb-4556-ba60-11a81566bdfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb\" (UID: \"60a7d133-03eb-4556-ba60-11a81566bdfc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:28.840559 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.840432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ltjr\" (UniqueName: \"kubernetes.io/projected/22fc2359-7150-4753-b04a-e4f618f47fb3-kube-api-access-6ltjr\") pod \"payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn\" (UID: \"22fc2359-7150-4753-b04a-e4f618f47fb3\") " pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:28.840559 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.840476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nrp8\" (UniqueName: \"kubernetes.io/projected/60a7d133-03eb-4556-ba60-11a81566bdfc-kube-api-access-4nrp8\") pod \"payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb\" (UID: \"60a7d133-03eb-4556-ba60-11a81566bdfc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:28.841225 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.841199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60a7d133-03eb-4556-ba60-11a81566bdfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb\" (UID: \"60a7d133-03eb-4556-ba60-11a81566bdfc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:28.848695 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.848666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nrp8\" (UniqueName: \"kubernetes.io/projected/60a7d133-03eb-4556-ba60-11a81566bdfc-kube-api-access-4nrp8\") pod \"payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb\" (UID: \"60a7d133-03eb-4556-ba60-11a81566bdfc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:28.941938 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.941799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fc2359-7150-4753-b04a-e4f618f47fb3-plugins-config-volume\") pod \"payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn\" (UID: \"22fc2359-7150-4753-b04a-e4f618f47fb3\") " pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:28.941938 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.941906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ltjr\" (UniqueName: \"kubernetes.io/projected/22fc2359-7150-4753-b04a-e4f618f47fb3-kube-api-access-6ltjr\") pod \"payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn\" (UID: \"22fc2359-7150-4753-b04a-e4f618f47fb3\") " pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:28.942833 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.942794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fc2359-7150-4753-b04a-e4f618f47fb3-plugins-config-volume\") pod \"payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn\" (UID: \"22fc2359-7150-4753-b04a-e4f618f47fb3\") " pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:28.951221 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.951192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ltjr\" (UniqueName: \"kubernetes.io/projected/22fc2359-7150-4753-b04a-e4f618f47fb3-kube-api-access-6ltjr\") pod \"payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn\" (UID: \"22fc2359-7150-4753-b04a-e4f618f47fb3\") " pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:28.963000 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:28.962972    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:29.016585 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.016511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:29.070215 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.070151    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:04:29.239246 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.239211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"]
Sep 09 13:04:29.241171 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:04:29.241147    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a7d133_03eb_4556_ba60_11a81566bdfc.slice/crio-c9dad4389ec6628338a59517fb1dad3d827cdf3f06d67d93c3aa5a27dbf9bfd4 WatchSource:0}: Error finding container c9dad4389ec6628338a59517fb1dad3d827cdf3f06d67d93c3aa5a27dbf9bfd4: Status 404 returned error can't find the container with id c9dad4389ec6628338a59517fb1dad3d827cdf3f06d67d93c3aa5a27dbf9bfd4
Sep 09 13:04:29.303425 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.297207    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"]
Sep 09 13:04:29.303499 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:04:29.303454    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fc2359_7150_4753_b04a_e4f618f47fb3.slice/crio-91c52c6083e3d4e03eddd702797939858cdd447471fc87500627791f2aaf823c WatchSource:0}: Error finding container 91c52c6083e3d4e03eddd702797939858cdd447471fc87500627791f2aaf823c: Status 404 returned error can't find the container with id 91c52c6083e3d4e03eddd702797939858cdd447471fc87500627791f2aaf823c
Sep 09 13:04:29.319226 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.319206    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:04:29.448176 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.448143    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-data\") pod \"5af351b4-6747-4692-83a0-bbb2db0a1a63\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") "
Sep 09 13:04:29.448176 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.448176    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-token\") pod \"5af351b4-6747-4692-83a0-bbb2db0a1a63\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") "
Sep 09 13:04:29.448379 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.448251    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-podinfo\") pod \"5af351b4-6747-4692-83a0-bbb2db0a1a63\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") "
Sep 09 13:04:29.448379 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.448301    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhcp6\" (UniqueName: \"kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-kube-api-access-bhcp6\") pod \"5af351b4-6747-4692-83a0-bbb2db0a1a63\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") "
Sep 09 13:04:29.448491 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.448442    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-data" (OuterVolumeSpecName: "istio-data") pod "5af351b4-6747-4692-83a0-bbb2db0a1a63" (UID: "5af351b4-6747-4692-83a0-bbb2db0a1a63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:04:29.448874 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.448813    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af351b4-6747-4692-83a0-bbb2db0a1a63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5af351b4-6747-4692-83a0-bbb2db0a1a63" (UID: "5af351b4-6747-4692-83a0-bbb2db0a1a63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:04:29.449065 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449008    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af351b4-6747-4692-83a0-bbb2db0a1a63-istiod-ca-cert\") pod \"5af351b4-6747-4692-83a0-bbb2db0a1a63\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") "
Sep 09 13:04:29.449173 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-socket\") pod \"5af351b4-6747-4692-83a0-bbb2db0a1a63\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") "
Sep 09 13:04:29.449173 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449169    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-credential-socket\") pod \"5af351b4-6747-4692-83a0-bbb2db0a1a63\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") "
Sep 09 13:04:29.449305 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-certs\") pod \"5af351b4-6747-4692-83a0-bbb2db0a1a63\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") "
Sep 09 13:04:29.449305 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449281    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-envoy\") pod \"5af351b4-6747-4692-83a0-bbb2db0a1a63\" (UID: \"5af351b4-6747-4692-83a0-bbb2db0a1a63\") "
Sep 09 13:04:29.449418 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5af351b4-6747-4692-83a0-bbb2db0a1a63" (UID: "5af351b4-6747-4692-83a0-bbb2db0a1a63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:04:29.449477 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449434    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5af351b4-6747-4692-83a0-bbb2db0a1a63" (UID: "5af351b4-6747-4692-83a0-bbb2db0a1a63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:04:29.449621 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5af351b4-6747-4692-83a0-bbb2db0a1a63" (UID: "5af351b4-6747-4692-83a0-bbb2db0a1a63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:04:29.449726 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449706    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-socket\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:29.449789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449733    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-credential-socket\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:29.449789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449749    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-workload-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:29.449789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449762    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-data\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:29.449789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.449774    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af351b4-6747-4692-83a0-bbb2db0a1a63-istiod-ca-cert\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:29.450806 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.450785    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-token" (OuterVolumeSpecName: "istio-token") pod "5af351b4-6747-4692-83a0-bbb2db0a1a63" (UID: "5af351b4-6747-4692-83a0-bbb2db0a1a63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:04:29.450893 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.450850    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-kube-api-access-bhcp6" (OuterVolumeSpecName: "kube-api-access-bhcp6") pod "5af351b4-6747-4692-83a0-bbb2db0a1a63" (UID: "5af351b4-6747-4692-83a0-bbb2db0a1a63"). InnerVolumeSpecName "kube-api-access-bhcp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:04:29.451079 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.451059    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5af351b4-6747-4692-83a0-bbb2db0a1a63" (UID: "5af351b4-6747-4692-83a0-bbb2db0a1a63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:04:29.451446 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.451430    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5af351b4-6747-4692-83a0-bbb2db0a1a63" (UID: "5af351b4-6747-4692-83a0-bbb2db0a1a63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:04:29.551283 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.551230    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-podinfo\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:29.551283 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.551252    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhcp6\" (UniqueName: \"kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-kube-api-access-bhcp6\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:29.551283 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.551261    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-envoy\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:29.551283 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:29.551269    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af351b4-6747-4692-83a0-bbb2db0a1a63-istio-token\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:30.002643 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.002609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb" event={"ID":"60a7d133-03eb-4556-ba60-11a81566bdfc","Type":"ContainerStarted","Data":"f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570"}
Sep 09 13:04:30.003062 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.002669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb" event={"ID":"60a7d133-03eb-4556-ba60-11a81566bdfc","Type":"ContainerStarted","Data":"c9dad4389ec6628338a59517fb1dad3d827cdf3f06d67d93c3aa5a27dbf9bfd4"}
Sep 09 13:04:30.003062 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.002699    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:30.004857 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.004607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn" event={"ID":"22fc2359-7150-4753-b04a-e4f618f47fb3","Type":"ContainerStarted","Data":"3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660"}
Sep 09 13:04:30.004857 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.004658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn" event={"ID":"22fc2359-7150-4753-b04a-e4f618f47fb3","Type":"ContainerStarted","Data":"91c52c6083e3d4e03eddd702797939858cdd447471fc87500627791f2aaf823c"}
Sep 09 13:04:30.004857 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.004701    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:30.006448 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.006423    2580 generic.go:356] "Generic (PLEG): container finished" podID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerID="2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f" exitCode=0
Sep 09 13:04:30.006555 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.006479    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"
Sep 09 13:04:30.006555 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.006505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" event={"ID":"5af351b4-6747-4692-83a0-bbb2db0a1a63","Type":"ContainerDied","Data":"2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f"}
Sep 09 13:04:30.006555 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.006541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw" event={"ID":"5af351b4-6747-4692-83a0-bbb2db0a1a63","Type":"ContainerDied","Data":"46340d1065d42eba21b475864ebf4a56c9947f195b48d3fc94a5d9941024558f"}
Sep 09 13:04:30.006658 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.006563    2580 scope.go:117] "RemoveContainer" containerID="2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f"
Sep 09 13:04:30.017743 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.017725    2580 scope.go:117] "RemoveContainer" containerID="69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b"
Sep 09 13:04:30.021107 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.021062    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb" podStartSLOduration=2.021046171 podStartE2EDuration="2.021046171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:04:30.019440767 +0000 UTC m=+952.939332129" watchObservedRunningTime="2026-09-09 13:04:30.021046171 +0000 UTC m=+952.940937509"
Sep 09 13:04:30.027088 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.027069    2580 scope.go:117] "RemoveContainer" containerID="2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f"
Sep 09 13:04:30.027381 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:04:30.027364    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f\": container with ID starting with 2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f not found: ID does not exist" containerID="2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f"
Sep 09 13:04:30.027438 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.027386    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f"} err="failed to get container status \"2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f\": rpc error: code = NotFound desc = could not find container \"2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f\": container with ID starting with 2f208e3458e62ea3033a1ff7711a3c1dc23e11d159babf438a662e505f47229f not found: ID does not exist"
Sep 09 13:04:30.027438 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.027402    2580 scope.go:117] "RemoveContainer" containerID="69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b"
Sep 09 13:04:30.027672 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:04:30.027655    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b\": container with ID starting with 69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b not found: ID does not exist" containerID="69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b"
Sep 09 13:04:30.027717 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.027676    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b"} err="failed to get container status \"69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b\": rpc error: code = NotFound desc = could not find container \"69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b\": container with ID starting with 69844ed8696d998afe12f40ca79afb903537d4c0406f03a9bf39615364079f1b not found: ID does not exist"
Sep 09 13:04:30.035671 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.035634    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn" podStartSLOduration=2.035623036 podStartE2EDuration="2.035623036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:04:30.034592213 +0000 UTC m=+952.954483554" watchObservedRunningTime="2026-09-09 13:04:30.035623036 +0000 UTC m=+952.955514375"
Sep 09 13:04:30.047974 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.047955    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"]
Sep 09 13:04:30.053871 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:30.053848    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-776d31f2-gw-openshift-default-84fdfc4684-78cfw"]
Sep 09 13:04:31.593618 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:31.592608    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" path="/var/lib/kubelet/pods/5af351b4-6747-4692-83a0-bbb2db0a1a63/volumes"
Sep 09 13:04:37.753932 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.753893    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"]
Sep 09 13:04:37.754358 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.754344    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy"
Sep 09 13:04:37.754401 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.754359    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy"
Sep 09 13:04:37.754401 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.754368    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy"
Sep 09 13:04:37.754401 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.754376    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy"
Sep 09 13:04:37.754501 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.754444    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy"
Sep 09 13:04:37.758969 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.758947    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.763056 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.762966    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b4ea80\"/\"kube-root-ca.crt\""
Sep 09 13:04:37.763548 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.763306    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b4ea80\"/\"default-dockercfg-d95d7\""
Sep 09 13:04:37.763548 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.763415    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b4ea80\"/\"autores-model-b4ea80-kserve-self-signed-certs\""
Sep 09 13:04:37.763548 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.763445    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b4ea80\"/\"openshift-service-ca.crt\""
Sep 09 13:04:37.766253 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.766229    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"]
Sep 09 13:04:37.822175 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.822067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz75p\" (UniqueName: \"kubernetes.io/projected/85b23f30-1152-445e-bf76-c5a45c394249-kube-api-access-cz75p\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.822175 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.822125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-model-cache\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.822380 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.822174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-tmp-dir\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.822380 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.822215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-dshm\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.822380 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.822256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-home\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.822380 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.822332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b23f30-1152-445e-bf76-c5a45c394249-tls-certs\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.923050 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.923011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-dshm\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.923246 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.923067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-home\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.923246 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.923118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b23f30-1152-445e-bf76-c5a45c394249-tls-certs\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.923246 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.923169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz75p\" (UniqueName: \"kubernetes.io/projected/85b23f30-1152-445e-bf76-c5a45c394249-kube-api-access-cz75p\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.923246 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.923194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-model-cache\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.923246 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.923214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-tmp-dir\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.923527 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.923503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-home\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.923584 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.923572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-model-cache\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.923584 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.923572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-tmp-dir\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.925418 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.925391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-dshm\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.925655 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.925638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b23f30-1152-445e-bf76-c5a45c394249-tls-certs\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:37.931119 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:37.931097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz75p\" (UniqueName: \"kubernetes.io/projected/85b23f30-1152-445e-bf76-c5a45c394249-kube-api-access-cz75p\") pod \"autores-model-b4ea80-kserve-75778776b7-n8w97\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") " pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:38.069545 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:38.069515    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:38.340777 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:38.340752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"]
Sep 09 13:04:38.343606 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:04:38.343583    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85b23f30_1152_445e_bf76_c5a45c394249.slice/crio-b6e1cad332d0d10b13cd59f43b8f9724eadd63cd7a9278a56398d9184326256c WatchSource:0}: Error finding container b6e1cad332d0d10b13cd59f43b8f9724eadd63cd7a9278a56398d9184326256c: Status 404 returned error can't find the container with id b6e1cad332d0d10b13cd59f43b8f9724eadd63cd7a9278a56398d9184326256c
Sep 09 13:04:39.053114 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:39.053082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97" event={"ID":"85b23f30-1152-445e-bf76-c5a45c394249","Type":"ContainerStarted","Data":"6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71"}
Sep 09 13:04:39.053457 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:39.053121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97" event={"ID":"85b23f30-1152-445e-bf76-c5a45c394249","Type":"ContainerStarted","Data":"b6e1cad332d0d10b13cd59f43b8f9724eadd63cd7a9278a56398d9184326256c"}
Sep 09 13:04:39.071100 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:39.071055    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97" podStartSLOduration=2.071040262 podStartE2EDuration="2.071040262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:04:39.069491313 +0000 UTC m=+961.989382655" watchObservedRunningTime="2026-09-09 13:04:39.071040262 +0000 UTC m=+961.990931600"
Sep 09 13:04:41.015445 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:41.015416    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:41.015878 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:41.015620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:44.166377 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:44.166345    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:04:48.069670 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:48.069637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:48.070062 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:48.069692    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:48.082981 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:48.082957    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:48.113006 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:48.112980    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:04:52.816041 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:52.816008    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"]
Sep 09 13:04:52.816485 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:52.816472    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af351b4-6747-4692-83a0-bbb2db0a1a63" containerName="istio-proxy"
Sep 09 13:04:52.821214 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:52.821196    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"
Sep 09 13:04:52.824280 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:52.824257    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t5rn5\""
Sep 09 13:04:52.824426 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:52.824406    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:04:52.825712 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:52.825688    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"]
Sep 09 13:04:52.955329 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:52.955290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e630ad-9f05-405d-8ce3-7deb45227406-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j\" (UID: \"33e630ad-9f05-405d-8ce3-7deb45227406\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"
Sep 09 13:04:53.056613 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:53.056585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e630ad-9f05-405d-8ce3-7deb45227406-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j\" (UID: \"33e630ad-9f05-405d-8ce3-7deb45227406\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"
Sep 09 13:04:53.057202 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:53.057184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e630ad-9f05-405d-8ce3-7deb45227406-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j\" (UID: \"33e630ad-9f05-405d-8ce3-7deb45227406\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"
Sep 09 13:04:53.133880 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:53.133806    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"
Sep 09 13:04:53.404287 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:53.404168    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"]
Sep 09 13:04:53.406414 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:04:53.406387    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e630ad_9f05_405d_8ce3_7deb45227406.slice/crio-64cbf3f2726f6a606ab9e5b9b36cda1b56ac268bdb2629ed3f4b01697ed078f1 WatchSource:0}: Error finding container 64cbf3f2726f6a606ab9e5b9b36cda1b56ac268bdb2629ed3f4b01697ed078f1: Status 404 returned error can't find the container with id 64cbf3f2726f6a606ab9e5b9b36cda1b56ac268bdb2629ed3f4b01697ed078f1
Sep 09 13:04:54.132997 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:54.132970    2580 generic.go:356] "Generic (PLEG): container finished" podID="33e630ad-9f05-405d-8ce3-7deb45227406" containerID="ed36de413e7cf317162ae663b6208ef1521510553e4fbe39f6da00bc5f82e0ff" exitCode=0
Sep 09 13:04:54.133350 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:54.133046    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j" event={"ID":"33e630ad-9f05-405d-8ce3-7deb45227406","Type":"ContainerDied","Data":"ed36de413e7cf317162ae663b6208ef1521510553e4fbe39f6da00bc5f82e0ff"}
Sep 09 13:04:54.133350 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:54.133094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j" event={"ID":"33e630ad-9f05-405d-8ce3-7deb45227406","Type":"ContainerStarted","Data":"64cbf3f2726f6a606ab9e5b9b36cda1b56ac268bdb2629ed3f4b01697ed078f1"}
Sep 09 13:04:55.269045 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:55.269021    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"
Sep 09 13:04:55.278585 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:55.278566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e630ad-9f05-405d-8ce3-7deb45227406-maas-api-service-ca\") pod \"33e630ad-9f05-405d-8ce3-7deb45227406\" (UID: \"33e630ad-9f05-405d-8ce3-7deb45227406\") "
Sep 09 13:04:55.278881 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:55.278860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e630ad-9f05-405d-8ce3-7deb45227406-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33e630ad-9f05-405d-8ce3-7deb45227406" (UID: "33e630ad-9f05-405d-8ce3-7deb45227406"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:04:55.379324 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:55.379291    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e630ad-9f05-405d-8ce3-7deb45227406-maas-api-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:56.144485 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:56.144454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j" event={"ID":"33e630ad-9f05-405d-8ce3-7deb45227406","Type":"ContainerDied","Data":"64cbf3f2726f6a606ab9e5b9b36cda1b56ac268bdb2629ed3f4b01697ed078f1"}
Sep 09 13:04:56.144661 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:56.144502    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64cbf3f2726f6a606ab9e5b9b36cda1b56ac268bdb2629ed3f4b01697ed078f1"
Sep 09 13:04:56.144661 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:56.144465    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"
Sep 09 13:04:58.694886 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:58.694791    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"]
Sep 09 13:04:58.695328 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:58.695056    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn" podUID="22fc2359-7150-4753-b04a-e4f618f47fb3" containerName="payload-processing" containerID="cri-o://3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660" gracePeriod=30
Sep 09 13:04:58.704384 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:58.704358    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"]
Sep 09 13:04:58.704592 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:58.704568    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb" podUID="60a7d133-03eb-4556-ba60-11a81566bdfc" containerName="payload-pre-processing" containerID="cri-o://f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570" gracePeriod=30
Sep 09 13:04:58.857948 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:58.857913    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"]
Sep 09 13:04:58.861902 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:58.861880    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0175d9d3-xq89j"]
Sep 09 13:04:58.959884 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:58.959856    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:58.970783 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:58.970765    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:59.008328 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.008307    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60a7d133-03eb-4556-ba60-11a81566bdfc-plugins-config-volume\") pod \"60a7d133-03eb-4556-ba60-11a81566bdfc\" (UID: \"60a7d133-03eb-4556-ba60-11a81566bdfc\") "
Sep 09 13:04:59.008455 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.008435    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nrp8\" (UniqueName: \"kubernetes.io/projected/60a7d133-03eb-4556-ba60-11a81566bdfc-kube-api-access-4nrp8\") pod \"60a7d133-03eb-4556-ba60-11a81566bdfc\" (UID: \"60a7d133-03eb-4556-ba60-11a81566bdfc\") "
Sep 09 13:04:59.008455 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.008452    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fc2359-7150-4753-b04a-e4f618f47fb3-plugins-config-volume\") pod \"22fc2359-7150-4753-b04a-e4f618f47fb3\" (UID: \"22fc2359-7150-4753-b04a-e4f618f47fb3\") "
Sep 09 13:04:59.008595 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.008478    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ltjr\" (UniqueName: \"kubernetes.io/projected/22fc2359-7150-4753-b04a-e4f618f47fb3-kube-api-access-6ltjr\") pod \"22fc2359-7150-4753-b04a-e4f618f47fb3\" (UID: \"22fc2359-7150-4753-b04a-e4f618f47fb3\") "
Sep 09 13:04:59.008732 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.008702    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a7d133-03eb-4556-ba60-11a81566bdfc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60a7d133-03eb-4556-ba60-11a81566bdfc" (UID: "60a7d133-03eb-4556-ba60-11a81566bdfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:04:59.008893 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.008871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fc2359-7150-4753-b04a-e4f618f47fb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22fc2359-7150-4753-b04a-e4f618f47fb3" (UID: "22fc2359-7150-4753-b04a-e4f618f47fb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:04:59.010615 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.010593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a7d133-03eb-4556-ba60-11a81566bdfc-kube-api-access-4nrp8" (OuterVolumeSpecName: "kube-api-access-4nrp8") pod "60a7d133-03eb-4556-ba60-11a81566bdfc" (UID: "60a7d133-03eb-4556-ba60-11a81566bdfc"). InnerVolumeSpecName "kube-api-access-4nrp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:04:59.010694 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.010654    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fc2359-7150-4753-b04a-e4f618f47fb3-kube-api-access-6ltjr" (OuterVolumeSpecName: "kube-api-access-6ltjr") pod "22fc2359-7150-4753-b04a-e4f618f47fb3" (UID: "22fc2359-7150-4753-b04a-e4f618f47fb3"). InnerVolumeSpecName "kube-api-access-6ltjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:04:59.109608 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.109575    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nrp8\" (UniqueName: \"kubernetes.io/projected/60a7d133-03eb-4556-ba60-11a81566bdfc-kube-api-access-4nrp8\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:59.109608 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.109602    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fc2359-7150-4753-b04a-e4f618f47fb3-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:59.109608 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.109611    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ltjr\" (UniqueName: \"kubernetes.io/projected/22fc2359-7150-4753-b04a-e4f618f47fb3-kube-api-access-6ltjr\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:59.109849 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.109620    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60a7d133-03eb-4556-ba60-11a81566bdfc-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:04:59.161698 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.161665    2580 generic.go:356] "Generic (PLEG): container finished" podID="22fc2359-7150-4753-b04a-e4f618f47fb3" containerID="3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660" exitCode=0
Sep 09 13:04:59.161877 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.161711    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"
Sep 09 13:04:59.161877 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.161752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn" event={"ID":"22fc2359-7150-4753-b04a-e4f618f47fb3","Type":"ContainerDied","Data":"3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660"}
Sep 09 13:04:59.161877 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.161799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn" event={"ID":"22fc2359-7150-4753-b04a-e4f618f47fb3","Type":"ContainerDied","Data":"91c52c6083e3d4e03eddd702797939858cdd447471fc87500627791f2aaf823c"}
Sep 09 13:04:59.161877 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.161847    2580 scope.go:117] "RemoveContainer" containerID="3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660"
Sep 09 13:04:59.163245 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.163223    2580 generic.go:356] "Generic (PLEG): container finished" podID="60a7d133-03eb-4556-ba60-11a81566bdfc" containerID="f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570" exitCode=0
Sep 09 13:04:59.163349 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.163267    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"
Sep 09 13:04:59.163349 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.163274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb" event={"ID":"60a7d133-03eb-4556-ba60-11a81566bdfc","Type":"ContainerDied","Data":"f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570"}
Sep 09 13:04:59.163349 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.163309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb" event={"ID":"60a7d133-03eb-4556-ba60-11a81566bdfc","Type":"ContainerDied","Data":"c9dad4389ec6628338a59517fb1dad3d827cdf3f06d67d93c3aa5a27dbf9bfd4"}
Sep 09 13:04:59.173171 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.173136    2580 scope.go:117] "RemoveContainer" containerID="3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660"
Sep 09 13:04:59.173524 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:04:59.173503    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660\": container with ID starting with 3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660 not found: ID does not exist" containerID="3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660"
Sep 09 13:04:59.173599 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.173529    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660"} err="failed to get container status \"3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660\": rpc error: code = NotFound desc = could not find container \"3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660\": container with ID starting with 3e0669f93d592fe995f3f6083753e2c87d3ebb82b79abb9dfcc1e4576ef81660 not found: ID does not exist"
Sep 09 13:04:59.173599 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.173545    2580 scope.go:117] "RemoveContainer" containerID="f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570"
Sep 09 13:04:59.182409 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.182388    2580 scope.go:117] "RemoveContainer" containerID="f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570"
Sep 09 13:04:59.182759 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:04:59.182737    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570\": container with ID starting with f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570 not found: ID does not exist" containerID="f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570"
Sep 09 13:04:59.182870 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.182764    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570"} err="failed to get container status \"f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570\": rpc error: code = NotFound desc = could not find container \"f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570\": container with ID starting with f674c25387a56af2787ec9c2ff84f1956ef4a736460f6f7edb7f561d289ac570 not found: ID does not exist"
Sep 09 13:04:59.184496 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.184477    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"]
Sep 09 13:04:59.190812 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.190792    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0175d9d3-6c4f9d9677-77jgn"]
Sep 09 13:04:59.200932 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.200889    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"]
Sep 09 13:04:59.204441 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.204423    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0175d9d3-5d77d8474-6ddsb"]
Sep 09 13:04:59.593434 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.593400    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22fc2359-7150-4753-b04a-e4f618f47fb3" path="/var/lib/kubelet/pods/22fc2359-7150-4753-b04a-e4f618f47fb3/volumes"
Sep 09 13:04:59.593758 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.593745    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e630ad-9f05-405d-8ce3-7deb45227406" path="/var/lib/kubelet/pods/33e630ad-9f05-405d-8ce3-7deb45227406/volumes"
Sep 09 13:04:59.594091 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:04:59.594079    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60a7d133-03eb-4556-ba60-11a81566bdfc" path="/var/lib/kubelet/pods/60a7d133-03eb-4556-ba60-11a81566bdfc/volumes"
Sep 09 13:05:02.868980 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:02.868944    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:05:04.269047 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.269012    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"]
Sep 09 13:05:04.269603 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.269582    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fc2359-7150-4753-b04a-e4f618f47fb3" containerName="payload-processing"
Sep 09 13:05:04.269688 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.269607    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fc2359-7150-4753-b04a-e4f618f47fb3" containerName="payload-processing"
Sep 09 13:05:04.269688 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.269640    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e630ad-9f05-405d-8ce3-7deb45227406" containerName="revoke-keys"
Sep 09 13:05:04.269688 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.269646    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e630ad-9f05-405d-8ce3-7deb45227406" containerName="revoke-keys"
Sep 09 13:05:04.269688 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.269674    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a7d133-03eb-4556-ba60-11a81566bdfc" containerName="payload-pre-processing"
Sep 09 13:05:04.269688 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.269680    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a7d133-03eb-4556-ba60-11a81566bdfc" containerName="payload-pre-processing"
Sep 09 13:05:04.269912 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.269757    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a7d133-03eb-4556-ba60-11a81566bdfc" containerName="payload-pre-processing"
Sep 09 13:05:04.269912 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.269765    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e630ad-9f05-405d-8ce3-7deb45227406" containerName="revoke-keys"
Sep 09 13:05:04.269912 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.269771    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="22fc2359-7150-4753-b04a-e4f618f47fb3" containerName="payload-processing"
Sep 09 13:05:04.273418 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.273390    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.276978 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.276952    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dbjk4\""
Sep 09 13:05:04.277901 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.277875    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9b2cda79-serving-cert\""
Sep 09 13:05:04.278159 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.278139    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 13:05:04.278437 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.278416    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:05:04.278684 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.278663    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 13:05:04.283755 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.283731    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"]
Sep 09 13:05:04.361880 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.361840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531cb7cf-c428-4c35-a17d-f30a0bb2749a-openshift-service-ca\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.362046 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.361894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-maas-api-tls\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.362046 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.361936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-metrics-certs\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.362046 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.362029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wvwn\" (UniqueName: \"kubernetes.io/projected/531cb7cf-c428-4c35-a17d-f30a0bb2749a-kube-api-access-2wvwn\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.463176 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.463140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-maas-api-tls\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.463370 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.463208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-metrics-certs\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.463370 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:05:04.463297    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-serving-cert: secret "maas-api-e2e-derive-9b2cda79-serving-cert" not found
Sep 09 13:05:04.463370 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.463319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wvwn\" (UniqueName: \"kubernetes.io/projected/531cb7cf-c428-4c35-a17d-f30a0bb2749a-kube-api-access-2wvwn\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.463545 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:05:04.463375    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-maas-api-tls podName:531cb7cf-c428-4c35-a17d-f30a0bb2749a nodeName:}" failed. No retries permitted until 2026-09-09 13:05:04.963353316 +0000 UTC m=+987.883244656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-maas-api-tls") pod "maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf" (UID: "531cb7cf-c428-4c35-a17d-f30a0bb2749a") : secret "maas-api-e2e-derive-9b2cda79-serving-cert" not found
Sep 09 13:05:04.463545 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.463411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531cb7cf-c428-4c35-a17d-f30a0bb2749a-openshift-service-ca\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.464036 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.464016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531cb7cf-c428-4c35-a17d-f30a0bb2749a-openshift-service-ca\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.465767 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.465741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-metrics-certs\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.471705 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.471681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wvwn\" (UniqueName: \"kubernetes.io/projected/531cb7cf-c428-4c35-a17d-f30a0bb2749a-kube-api-access-2wvwn\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.883417 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.883378    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"]
Sep 09 13:05:04.888422 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.888389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:04.891453 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.891425    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9b2cda79-dockercfg-ff85x\""
Sep 09 13:05:04.892099 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.892080    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9b2cda79\""
Sep 09 13:05:04.897403 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.897370    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"]
Sep 09 13:05:04.968450 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.968415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5kvq\" (UniqueName: \"kubernetes.io/projected/dd648616-1d8b-450d-935c-a7ac61cd39d9-kube-api-access-s5kvq\") pod \"payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf\" (UID: \"dd648616-1d8b-450d-935c-a7ac61cd39d9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:04.968641 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.968560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-maas-api-tls\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:04.968641 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.968602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd648616-1d8b-450d-935c-a7ac61cd39d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf\" (UID: \"dd648616-1d8b-450d-935c-a7ac61cd39d9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:04.971319 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:04.971291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-maas-api-tls\") pod \"maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:05.069988 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:05.069945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kvq\" (UniqueName: \"kubernetes.io/projected/dd648616-1d8b-450d-935c-a7ac61cd39d9-kube-api-access-s5kvq\") pod \"payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf\" (UID: \"dd648616-1d8b-450d-935c-a7ac61cd39d9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:05.070194 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:05.070037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd648616-1d8b-450d-935c-a7ac61cd39d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf\" (UID: \"dd648616-1d8b-450d-935c-a7ac61cd39d9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:05.070671 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:05.070651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd648616-1d8b-450d-935c-a7ac61cd39d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf\" (UID: \"dd648616-1d8b-450d-935c-a7ac61cd39d9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:05.081922 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:05.081891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5kvq\" (UniqueName: \"kubernetes.io/projected/dd648616-1d8b-450d-935c-a7ac61cd39d9-kube-api-access-s5kvq\") pod \"payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf\" (UID: \"dd648616-1d8b-450d-935c-a7ac61cd39d9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:05.190346 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:05.190253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:05.199973 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:05.199944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:05.466149 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:05.466008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"]
Sep 09 13:05:05.468643 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:05:05.468618    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531cb7cf_c428_4c35_a17d_f30a0bb2749a.slice/crio-969bf512b856974e7aa5741e612cf5ff4bf6a11c6adbb85c61c0e100ebe26869 WatchSource:0}: Error finding container 969bf512b856974e7aa5741e612cf5ff4bf6a11c6adbb85c61c0e100ebe26869: Status 404 returned error can't find the container with id 969bf512b856974e7aa5741e612cf5ff4bf6a11c6adbb85c61c0e100ebe26869
Sep 09 13:05:05.470122 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:05.470105    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 13:05:05.477185 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:05.477148    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"]
Sep 09 13:05:05.479415 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:05:05.479391    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd648616_1d8b_450d_935c_a7ac61cd39d9.slice/crio-6adb292644c05c8cd0edb8867623e42940b202b85b90627222e1628f712f79f7 WatchSource:0}: Error finding container 6adb292644c05c8cd0edb8867623e42940b202b85b90627222e1628f712f79f7: Status 404 returned error can't find the container with id 6adb292644c05c8cd0edb8867623e42940b202b85b90627222e1628f712f79f7
Sep 09 13:05:06.205150 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:06.205052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf" event={"ID":"dd648616-1d8b-450d-935c-a7ac61cd39d9","Type":"ContainerStarted","Data":"dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7"}
Sep 09 13:05:06.205150 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:06.205117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf" event={"ID":"dd648616-1d8b-450d-935c-a7ac61cd39d9","Type":"ContainerStarted","Data":"6adb292644c05c8cd0edb8867623e42940b202b85b90627222e1628f712f79f7"}
Sep 09 13:05:06.205379 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:06.205215    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:06.206785 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:06.206762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf" event={"ID":"531cb7cf-c428-4c35-a17d-f30a0bb2749a","Type":"ContainerStarted","Data":"44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c"}
Sep 09 13:05:06.206931 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:06.206809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf" event={"ID":"531cb7cf-c428-4c35-a17d-f30a0bb2749a","Type":"ContainerStarted","Data":"969bf512b856974e7aa5741e612cf5ff4bf6a11c6adbb85c61c0e100ebe26869"}
Sep 09 13:05:06.206931 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:06.206867    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:06.236015 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:06.235966    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf" podStartSLOduration=2.235950638 podStartE2EDuration="2.235950638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:05:06.219664415 +0000 UTC m=+989.139555754" watchObservedRunningTime="2026-09-09 13:05:06.235950638 +0000 UTC m=+989.155841980"
Sep 09 13:05:06.236266 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:06.236246    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf" podStartSLOduration=1.8062866560000002 podStartE2EDuration="2.236240247s" totalImagesPullingTime="429.953591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:05:06.234406457 +0000 UTC m=+989.154297813" watchObservedRunningTime="2026-09-09 13:05:06.236240247 +0000 UTC m=+989.156131585"
Sep 09 13:05:10.362100 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:10.362065    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"]
Sep 09 13:05:10.365476 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:10.365459    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"
Sep 09 13:05:10.367958 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:10.367939    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t5rn5\""
Sep 09 13:05:10.372577 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:10.372552    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"]
Sep 09 13:05:10.427051 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:10.427016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9a4cd-9072-40b6-82bb-6bb114640fce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f\" (UID: \"e8a9a4cd-9072-40b6-82bb-6bb114640fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"
Sep 09 13:05:10.528568 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:10.528532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9a4cd-9072-40b6-82bb-6bb114640fce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f\" (UID: \"e8a9a4cd-9072-40b6-82bb-6bb114640fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"
Sep 09 13:05:10.529132 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:10.529113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9a4cd-9072-40b6-82bb-6bb114640fce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f\" (UID: \"e8a9a4cd-9072-40b6-82bb-6bb114640fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"
Sep 09 13:05:10.677251 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:10.677158    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"
Sep 09 13:05:10.947838 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:10.947799    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"]
Sep 09 13:05:10.949676 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:05:10.949645    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a9a4cd_9072_40b6_82bb_6bb114640fce.slice/crio-9af5ee43059ee49500be94031e2a5ae7ce30d8adc554e7aed7ae192596792b87 WatchSource:0}: Error finding container 9af5ee43059ee49500be94031e2a5ae7ce30d8adc554e7aed7ae192596792b87: Status 404 returned error can't find the container with id 9af5ee43059ee49500be94031e2a5ae7ce30d8adc554e7aed7ae192596792b87
Sep 09 13:05:11.236336 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:11.236300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f" event={"ID":"e8a9a4cd-9072-40b6-82bb-6bb114640fce","Type":"ContainerStarted","Data":"da2ca6ae271d44d903fabd8c7611b2187871598fe1524eda1d746000d4a9a394"}
Sep 09 13:05:11.236508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:11.236361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f" event={"ID":"e8a9a4cd-9072-40b6-82bb-6bb114640fce","Type":"ContainerStarted","Data":"9af5ee43059ee49500be94031e2a5ae7ce30d8adc554e7aed7ae192596792b87"}
Sep 09 13:05:11.250407 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:11.250356    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f" podStartSLOduration=1.250339353 podStartE2EDuration="1.250339353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:05:11.24979655 +0000 UTC m=+994.169687890" watchObservedRunningTime="2026-09-09 13:05:11.250339353 +0000 UTC m=+994.170230770"
Sep 09 13:05:12.218143 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:12.218111    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:12.244525 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:12.244476    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8a9a4cd-9072-40b6-82bb-6bb114640fce" containerID="da2ca6ae271d44d903fabd8c7611b2187871598fe1524eda1d746000d4a9a394" exitCode=0
Sep 09 13:05:12.244525 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:12.244520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f" event={"ID":"e8a9a4cd-9072-40b6-82bb-6bb114640fce","Type":"ContainerDied","Data":"da2ca6ae271d44d903fabd8c7611b2187871598fe1524eda1d746000d4a9a394"}
Sep 09 13:05:13.388173 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:13.388142    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"
Sep 09 13:05:13.456051 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:13.456013    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9a4cd-9072-40b6-82bb-6bb114640fce-maas-api-service-ca\") pod \"e8a9a4cd-9072-40b6-82bb-6bb114640fce\" (UID: \"e8a9a4cd-9072-40b6-82bb-6bb114640fce\") "
Sep 09 13:05:13.456399 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:13.456373    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a9a4cd-9072-40b6-82bb-6bb114640fce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8a9a4cd-9072-40b6-82bb-6bb114640fce" (UID: "e8a9a4cd-9072-40b6-82bb-6bb114640fce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:05:13.557076 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:13.557043    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9a4cd-9072-40b6-82bb-6bb114640fce-maas-api-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:14.258422 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:14.258395    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"
Sep 09 13:05:14.258422 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:14.258401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f" event={"ID":"e8a9a4cd-9072-40b6-82bb-6bb114640fce","Type":"ContainerDied","Data":"9af5ee43059ee49500be94031e2a5ae7ce30d8adc554e7aed7ae192596792b87"}
Sep 09 13:05:14.258697 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:14.258446    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9af5ee43059ee49500be94031e2a5ae7ce30d8adc554e7aed7ae192596792b87"
Sep 09 13:05:17.214377 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:17.214348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:17.863529 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:17.863494    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:05:21.007036 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:21.006998    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"]
Sep 09 13:05:21.010025 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:21.009998    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4ea80-qnb5f"]
Sep 09 13:05:21.593105 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:21.593068    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a9a4cd-9072-40b6-82bb-6bb114640fce" path="/var/lib/kubelet/pods/e8a9a4cd-9072-40b6-82bb-6bb114640fce/volumes"
Sep 09 13:05:26.383882 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:26.383844    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"]
Sep 09 13:05:28.118356 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.118318    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"]
Sep 09 13:05:28.118745 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.118732    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a9a4cd-9072-40b6-82bb-6bb114640fce" containerName="revoke-keys"
Sep 09 13:05:28.118794 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.118747    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a9a4cd-9072-40b6-82bb-6bb114640fce" containerName="revoke-keys"
Sep 09 13:05:28.118867 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.118857    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a9a4cd-9072-40b6-82bb-6bb114640fce" containerName="revoke-keys"
Sep 09 13:05:28.123518 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.123500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"
Sep 09 13:05:28.126107 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.126084    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t5rn5\""
Sep 09 13:05:28.130668 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.130639    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"]
Sep 09 13:05:28.309146 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.309104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823abea9-6d19-4a47-823b-5e84aa01ecb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2\" (UID: \"823abea9-6d19-4a47-823b-5e84aa01ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"
Sep 09 13:05:28.410073 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.409983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823abea9-6d19-4a47-823b-5e84aa01ecb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2\" (UID: \"823abea9-6d19-4a47-823b-5e84aa01ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"
Sep 09 13:05:28.410612 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.410593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823abea9-6d19-4a47-823b-5e84aa01ecb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2\" (UID: \"823abea9-6d19-4a47-823b-5e84aa01ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"
Sep 09 13:05:28.436814 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.436787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"
Sep 09 13:05:28.707869 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:28.707843    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"]
Sep 09 13:05:28.709376 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:05:28.709341    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823abea9_6d19_4a47_823b_5e84aa01ecb1.slice/crio-3592e8e5f70b1d00a9fb2d39ef71d7f1c190a66e2e1c1c94e01ddcf824882227 WatchSource:0}: Error finding container 3592e8e5f70b1d00a9fb2d39ef71d7f1c190a66e2e1c1c94e01ddcf824882227: Status 404 returned error can't find the container with id 3592e8e5f70b1d00a9fb2d39ef71d7f1c190a66e2e1c1c94e01ddcf824882227
Sep 09 13:05:29.341031 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:29.340991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2" event={"ID":"823abea9-6d19-4a47-823b-5e84aa01ecb1","Type":"ContainerStarted","Data":"6dc0021d538580adfac856d23be276a8c971f62034834d6293eb6fef20be06e6"}
Sep 09 13:05:29.341483 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:29.341054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2" event={"ID":"823abea9-6d19-4a47-823b-5e84aa01ecb1","Type":"ContainerStarted","Data":"3592e8e5f70b1d00a9fb2d39ef71d7f1c190a66e2e1c1c94e01ddcf824882227"}
Sep 09 13:05:29.356987 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:29.356931    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2" podStartSLOduration=1.356916437 podStartE2EDuration="1.356916437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:05:29.355005215 +0000 UTC m=+1012.274896553" watchObservedRunningTime="2026-09-09 13:05:29.356916437 +0000 UTC m=+1012.276807775"
Sep 09 13:05:30.348203 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:30.348167    2580 generic.go:356] "Generic (PLEG): container finished" podID="823abea9-6d19-4a47-823b-5e84aa01ecb1" containerID="6dc0021d538580adfac856d23be276a8c971f62034834d6293eb6fef20be06e6" exitCode=0
Sep 09 13:05:30.348613 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:30.348248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2" event={"ID":"823abea9-6d19-4a47-823b-5e84aa01ecb1","Type":"ContainerDied","Data":"6dc0021d538580adfac856d23be276a8c971f62034834d6293eb6fef20be06e6"}
Sep 09 13:05:31.488657 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:31.488630    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"
Sep 09 13:05:31.644358 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:31.644268    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823abea9-6d19-4a47-823b-5e84aa01ecb1-maas-api-service-ca\") pod \"823abea9-6d19-4a47-823b-5e84aa01ecb1\" (UID: \"823abea9-6d19-4a47-823b-5e84aa01ecb1\") "
Sep 09 13:05:31.644617 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:31.644593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823abea9-6d19-4a47-823b-5e84aa01ecb1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "823abea9-6d19-4a47-823b-5e84aa01ecb1" (UID: "823abea9-6d19-4a47-823b-5e84aa01ecb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:05:31.745863 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:31.745807    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823abea9-6d19-4a47-823b-5e84aa01ecb1-maas-api-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:32.176782 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:32.176745    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:05:32.360457 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:32.360424    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"
Sep 09 13:05:32.360626 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:32.360468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2" event={"ID":"823abea9-6d19-4a47-823b-5e84aa01ecb1","Type":"ContainerDied","Data":"3592e8e5f70b1d00a9fb2d39ef71d7f1c190a66e2e1c1c94e01ddcf824882227"}
Sep 09 13:05:32.360626 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:32.360515    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3592e8e5f70b1d00a9fb2d39ef71d7f1c190a66e2e1c1c94e01ddcf824882227"
Sep 09 13:05:34.765700 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:34.765668    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"]
Sep 09 13:05:34.766142 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:34.765884    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf" podUID="531cb7cf-c428-4c35-a17d-f30a0bb2749a" containerName="maas-api" containerID="cri-o://44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c" gracePeriod=30
Sep 09 13:05:34.817669 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:34.817634    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"]
Sep 09 13:05:34.818281 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:34.818210    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf" podUID="dd648616-1d8b-450d-935c-a7ac61cd39d9" containerName="payload-pre-processing" containerID="cri-o://dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7" gracePeriod=30
Sep 09 13:05:34.964058 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:34.964018    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"]
Sep 09 13:05:34.966347 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:34.966320    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b2cda79-5qkl2"]
Sep 09 13:05:35.074067 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.074041    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:35.114522 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.114495    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:35.178583 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.178542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd648616-1d8b-450d-935c-a7ac61cd39d9-plugins-config-volume\") pod \"dd648616-1d8b-450d-935c-a7ac61cd39d9\" (UID: \"dd648616-1d8b-450d-935c-a7ac61cd39d9\") "
Sep 09 13:05:35.178758 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.178619    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wvwn\" (UniqueName: \"kubernetes.io/projected/531cb7cf-c428-4c35-a17d-f30a0bb2749a-kube-api-access-2wvwn\") pod \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") "
Sep 09 13:05:35.178758 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.178695    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5kvq\" (UniqueName: \"kubernetes.io/projected/dd648616-1d8b-450d-935c-a7ac61cd39d9-kube-api-access-s5kvq\") pod \"dd648616-1d8b-450d-935c-a7ac61cd39d9\" (UID: \"dd648616-1d8b-450d-935c-a7ac61cd39d9\") "
Sep 09 13:05:35.178758 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.178731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-maas-api-tls\") pod \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") "
Sep 09 13:05:35.178988 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.178760    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531cb7cf-c428-4c35-a17d-f30a0bb2749a-openshift-service-ca\") pod \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") "
Sep 09 13:05:35.178988 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.178802    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-metrics-certs\") pod \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\" (UID: \"531cb7cf-c428-4c35-a17d-f30a0bb2749a\") "
Sep 09 13:05:35.179082 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.178979    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd648616-1d8b-450d-935c-a7ac61cd39d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd648616-1d8b-450d-935c-a7ac61cd39d9" (UID: "dd648616-1d8b-450d-935c-a7ac61cd39d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:05:35.179391 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.179339    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531cb7cf-c428-4c35-a17d-f30a0bb2749a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "531cb7cf-c428-4c35-a17d-f30a0bb2749a" (UID: "531cb7cf-c428-4c35-a17d-f30a0bb2749a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:05:35.179391 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.179344    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd648616-1d8b-450d-935c-a7ac61cd39d9-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:35.181068 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.181045    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "531cb7cf-c428-4c35-a17d-f30a0bb2749a" (UID: "531cb7cf-c428-4c35-a17d-f30a0bb2749a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:05:35.181068 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.181046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd648616-1d8b-450d-935c-a7ac61cd39d9-kube-api-access-s5kvq" (OuterVolumeSpecName: "kube-api-access-s5kvq") pod "dd648616-1d8b-450d-935c-a7ac61cd39d9" (UID: "dd648616-1d8b-450d-935c-a7ac61cd39d9"). InnerVolumeSpecName "kube-api-access-s5kvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:05:35.181465 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.181449    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "531cb7cf-c428-4c35-a17d-f30a0bb2749a" (UID: "531cb7cf-c428-4c35-a17d-f30a0bb2749a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:05:35.181506 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.181464    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531cb7cf-c428-4c35-a17d-f30a0bb2749a-kube-api-access-2wvwn" (OuterVolumeSpecName: "kube-api-access-2wvwn") pod "531cb7cf-c428-4c35-a17d-f30a0bb2749a" (UID: "531cb7cf-c428-4c35-a17d-f30a0bb2749a"). InnerVolumeSpecName "kube-api-access-2wvwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:05:35.279957 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.279914    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wvwn\" (UniqueName: \"kubernetes.io/projected/531cb7cf-c428-4c35-a17d-f30a0bb2749a-kube-api-access-2wvwn\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:35.279957 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.279948    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5kvq\" (UniqueName: \"kubernetes.io/projected/dd648616-1d8b-450d-935c-a7ac61cd39d9-kube-api-access-s5kvq\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:35.279957 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.279960    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-maas-api-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:35.280207 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.279972    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531cb7cf-c428-4c35-a17d-f30a0bb2749a-openshift-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:35.280207 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.279983    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531cb7cf-c428-4c35-a17d-f30a0bb2749a-metrics-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:35.377480 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.377439    2580 generic.go:356] "Generic (PLEG): container finished" podID="dd648616-1d8b-450d-935c-a7ac61cd39d9" containerID="dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7" exitCode=0
Sep 09 13:05:35.377684 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.377503    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"
Sep 09 13:05:35.377684 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.377519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf" event={"ID":"dd648616-1d8b-450d-935c-a7ac61cd39d9","Type":"ContainerDied","Data":"dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7"}
Sep 09 13:05:35.377684 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.377616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf" event={"ID":"dd648616-1d8b-450d-935c-a7ac61cd39d9","Type":"ContainerDied","Data":"6adb292644c05c8cd0edb8867623e42940b202b85b90627222e1628f712f79f7"}
Sep 09 13:05:35.377684 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.377651    2580 scope.go:117] "RemoveContainer" containerID="dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7"
Sep 09 13:05:35.379146 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.379121    2580 generic.go:356] "Generic (PLEG): container finished" podID="531cb7cf-c428-4c35-a17d-f30a0bb2749a" containerID="44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c" exitCode=0
Sep 09 13:05:35.379273 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.379177    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"
Sep 09 13:05:35.379273 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.379195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf" event={"ID":"531cb7cf-c428-4c35-a17d-f30a0bb2749a","Type":"ContainerDied","Data":"44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c"}
Sep 09 13:05:35.379273 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.379238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf" event={"ID":"531cb7cf-c428-4c35-a17d-f30a0bb2749a","Type":"ContainerDied","Data":"969bf512b856974e7aa5741e612cf5ff4bf6a11c6adbb85c61c0e100ebe26869"}
Sep 09 13:05:35.388039 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.388014    2580 scope.go:117] "RemoveContainer" containerID="dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7"
Sep 09 13:05:35.388428 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:05:35.388396    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7\": container with ID starting with dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7 not found: ID does not exist" containerID="dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7"
Sep 09 13:05:35.388486 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.388421    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7"} err="failed to get container status \"dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7\": rpc error: code = NotFound desc = could not find container \"dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7\": container with ID starting with dedf8ef3c6369669445509081bfec6d3b105f8c08571afe378e7c74c9aad17c7 not found: ID does not exist"
Sep 09 13:05:35.388486 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.388439    2580 scope.go:117] "RemoveContainer" containerID="44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c"
Sep 09 13:05:35.397441 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.397423    2580 scope.go:117] "RemoveContainer" containerID="44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c"
Sep 09 13:05:35.397722 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:05:35.397705    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c\": container with ID starting with 44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c not found: ID does not exist" containerID="44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c"
Sep 09 13:05:35.397786 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.397727    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c"} err="failed to get container status \"44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c\": rpc error: code = NotFound desc = could not find container \"44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c\": container with ID starting with 44e10dd91cf171d622d72aba251b8c3efebd2607995279ccc6a1522b058d042c not found: ID does not exist"
Sep 09 13:05:35.401645 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.401624    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"]
Sep 09 13:05:35.405474 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.405451    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b2cda79-5cfbd5fcdf-tjdgf"]
Sep 09 13:05:35.415112 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.415087    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"]
Sep 09 13:05:35.418936 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.418914    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b2cda79-54d4868874-th5xf"]
Sep 09 13:05:35.591496 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.591450    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="531cb7cf-c428-4c35-a17d-f30a0bb2749a" path="/var/lib/kubelet/pods/531cb7cf-c428-4c35-a17d-f30a0bb2749a/volumes"
Sep 09 13:05:35.591953 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.591933    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823abea9-6d19-4a47-823b-5e84aa01ecb1" path="/var/lib/kubelet/pods/823abea9-6d19-4a47-823b-5e84aa01ecb1/volumes"
Sep 09 13:05:35.592336 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:35.592317    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd648616-1d8b-450d-935c-a7ac61cd39d9" path="/var/lib/kubelet/pods/dd648616-1d8b-450d-935c-a7ac61cd39d9/volumes"
Sep 09 13:05:41.406710 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.406643    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97" podUID="85b23f30-1152-445e-bf76-c5a45c394249" containerName="main" containerID="cri-o://6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71" gracePeriod=45
Sep 09 13:05:41.662393 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.662324    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:05:41.741475 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.741444    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz75p\" (UniqueName: \"kubernetes.io/projected/85b23f30-1152-445e-bf76-c5a45c394249-kube-api-access-cz75p\") pod \"85b23f30-1152-445e-bf76-c5a45c394249\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") "
Sep 09 13:05:41.741657 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.741542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-home\") pod \"85b23f30-1152-445e-bf76-c5a45c394249\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") "
Sep 09 13:05:41.741657 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.741612    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-dshm\") pod \"85b23f30-1152-445e-bf76-c5a45c394249\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") "
Sep 09 13:05:41.741657 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.741653    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-model-cache\") pod \"85b23f30-1152-445e-bf76-c5a45c394249\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") "
Sep 09 13:05:41.741836 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.741719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-tmp-dir\") pod \"85b23f30-1152-445e-bf76-c5a45c394249\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") "
Sep 09 13:05:41.741836 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.741747    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b23f30-1152-445e-bf76-c5a45c394249-tls-certs\") pod \"85b23f30-1152-445e-bf76-c5a45c394249\" (UID: \"85b23f30-1152-445e-bf76-c5a45c394249\") "
Sep 09 13:05:41.741962 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.741894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-home" (OuterVolumeSpecName: "home") pod "85b23f30-1152-445e-bf76-c5a45c394249" (UID: "85b23f30-1152-445e-bf76-c5a45c394249"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:05:41.742014 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.741975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85b23f30-1152-445e-bf76-c5a45c394249" (UID: "85b23f30-1152-445e-bf76-c5a45c394249"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:05:41.742014 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.741982    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-model-cache" (OuterVolumeSpecName: "model-cache") pod "85b23f30-1152-445e-bf76-c5a45c394249" (UID: "85b23f30-1152-445e-bf76-c5a45c394249"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:05:41.742140 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.742126    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-home\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:41.742234 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.742142    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-model-cache\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:41.742234 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.742153    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-tmp-dir\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:41.743908 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.743882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b23f30-1152-445e-bf76-c5a45c394249-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85b23f30-1152-445e-bf76-c5a45c394249" (UID: "85b23f30-1152-445e-bf76-c5a45c394249"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:05:41.744011 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.743935    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b23f30-1152-445e-bf76-c5a45c394249-kube-api-access-cz75p" (OuterVolumeSpecName: "kube-api-access-cz75p") pod "85b23f30-1152-445e-bf76-c5a45c394249" (UID: "85b23f30-1152-445e-bf76-c5a45c394249"). InnerVolumeSpecName "kube-api-access-cz75p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:05:41.744011 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.743998    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-dshm" (OuterVolumeSpecName: "dshm") pod "85b23f30-1152-445e-bf76-c5a45c394249" (UID: "85b23f30-1152-445e-bf76-c5a45c394249"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:05:41.843351 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.843320    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b23f30-1152-445e-bf76-c5a45c394249-dshm\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:41.843351 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.843347    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b23f30-1152-445e-bf76-c5a45c394249-tls-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:41.843351 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:41.843358    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz75p\" (UniqueName: \"kubernetes.io/projected/85b23f30-1152-445e-bf76-c5a45c394249-kube-api-access-cz75p\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:05:42.420136 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:42.420093    2580 generic.go:356] "Generic (PLEG): container finished" podID="85b23f30-1152-445e-bf76-c5a45c394249" containerID="6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71" exitCode=0
Sep 09 13:05:42.420573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:42.420146    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"
Sep 09 13:05:42.420573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:42.420182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97" event={"ID":"85b23f30-1152-445e-bf76-c5a45c394249","Type":"ContainerDied","Data":"6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71"}
Sep 09 13:05:42.420573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:42.420226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97" event={"ID":"85b23f30-1152-445e-bf76-c5a45c394249","Type":"ContainerDied","Data":"b6e1cad332d0d10b13cd59f43b8f9724eadd63cd7a9278a56398d9184326256c"}
Sep 09 13:05:42.420573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:42.420244    2580 scope.go:117] "RemoveContainer" containerID="6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71"
Sep 09 13:05:42.430268 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:42.430248    2580 scope.go:117] "RemoveContainer" containerID="6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71"
Sep 09 13:05:42.430592 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:05:42.430568    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71\": container with ID starting with 6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71 not found: ID does not exist" containerID="6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71"
Sep 09 13:05:42.430657 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:42.430601    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71"} err="failed to get container status \"6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71\": rpc error: code = NotFound desc = could not find container \"6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71\": container with ID starting with 6c12d7a5da55f1702cf65c4f488b8b6ad24d299e02e0601d5aeeb1c1c975bf71 not found: ID does not exist"
Sep 09 13:05:42.442638 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:42.442602    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"]
Sep 09 13:05:42.445951 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:42.445928    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-b4ea80/autores-model-b4ea80-kserve-75778776b7-n8w97"]
Sep 09 13:05:43.592528 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:43.592484    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85b23f30-1152-445e-bf76-c5a45c394249" path="/var/lib/kubelet/pods/85b23f30-1152-445e-bf76-c5a45c394249/volumes"
Sep 09 13:05:47.562461 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:47.562423    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:05:57.169139 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169099    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"]
Sep 09 13:05:57.169526 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169513    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b23f30-1152-445e-bf76-c5a45c394249" containerName="main"
Sep 09 13:05:57.169526 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169525    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b23f30-1152-445e-bf76-c5a45c394249" containerName="main"
Sep 09 13:05:57.169596 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169539    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823abea9-6d19-4a47-823b-5e84aa01ecb1" containerName="revoke-keys"
Sep 09 13:05:57.169596 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169545    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="823abea9-6d19-4a47-823b-5e84aa01ecb1" containerName="revoke-keys"
Sep 09 13:05:57.169596 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169559    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd648616-1d8b-450d-935c-a7ac61cd39d9" containerName="payload-pre-processing"
Sep 09 13:05:57.169596 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169565    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd648616-1d8b-450d-935c-a7ac61cd39d9" containerName="payload-pre-processing"
Sep 09 13:05:57.169596 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169573    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="531cb7cf-c428-4c35-a17d-f30a0bb2749a" containerName="maas-api"
Sep 09 13:05:57.169596 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169578    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="531cb7cf-c428-4c35-a17d-f30a0bb2749a" containerName="maas-api"
Sep 09 13:05:57.169777 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169634    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="823abea9-6d19-4a47-823b-5e84aa01ecb1" containerName="revoke-keys"
Sep 09 13:05:57.169777 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169647    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b23f30-1152-445e-bf76-c5a45c394249" containerName="main"
Sep 09 13:05:57.169777 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169655    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd648616-1d8b-450d-935c-a7ac61cd39d9" containerName="payload-pre-processing"
Sep 09 13:05:57.169777 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.169661    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="531cb7cf-c428-4c35-a17d-f30a0bb2749a" containerName="maas-api"
Sep 09 13:05:57.174137 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.174122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.176593 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.176574    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-a333cccb-kserve-self-signed-certs\""
Sep 09 13:05:57.187147 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.187118    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"]
Sep 09 13:05:57.300458 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.300417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-dshm\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.300458 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.300458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tls-certs\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.300724 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.300485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-home\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.300724 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.300556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-model-cache\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.300724 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.300594    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hms72\" (UniqueName: \"kubernetes.io/projected/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-kube-api-access-hms72\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.300724 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.300660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tmp-dir\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.401727 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.401679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-model-cache\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.401727 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.401731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hms72\" (UniqueName: \"kubernetes.io/projected/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-kube-api-access-hms72\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.402032 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.401864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tmp-dir\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.402032 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.401981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-dshm\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.402032 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.402000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tls-certs\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.402032 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.402018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-home\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.402249 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.402182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-model-cache\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.402287 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.402244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tmp-dir\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.402379 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.402359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-home\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.404453 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.404426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-dshm\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.404651 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.404632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tls-certs\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.410065 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.410041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hms72\" (UniqueName: \"kubernetes.io/projected/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-kube-api-access-hms72\") pod \"e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") " pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.488729 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.488630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:05:57.762885 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:57.762862    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"]
Sep 09 13:05:57.765014 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:05:57.764982    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f7bd5eb_98b5_4606_9da2_66c6bfa91ef2.slice/crio-40ffc9667a6fa7f19229afd115d760708debdfea0fd8d2b82935a79ac0f68344 WatchSource:0}: Error finding container 40ffc9667a6fa7f19229afd115d760708debdfea0fd8d2b82935a79ac0f68344: Status 404 returned error can't find the container with id 40ffc9667a6fa7f19229afd115d760708debdfea0fd8d2b82935a79ac0f68344
Sep 09 13:05:58.507998 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:58.507960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2" event={"ID":"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2","Type":"ContainerStarted","Data":"8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103"}
Sep 09 13:05:58.507998 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:58.508005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2" event={"ID":"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2","Type":"ContainerStarted","Data":"40ffc9667a6fa7f19229afd115d760708debdfea0fd8d2b82935a79ac0f68344"}
Sep 09 13:05:58.525051 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:05:58.524999    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2" podStartSLOduration=1.524985947 podStartE2EDuration="1.524985947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:05:58.523811001 +0000 UTC m=+1041.443702356" watchObservedRunningTime="2026-09-09 13:05:58.524985947 +0000 UTC m=+1041.444877286"
Sep 09 13:06:01.673681 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:01.673640    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:06:07.489335 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:07.489297    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:06:07.489777 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:07.489362    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:06:07.502455 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:07.502425    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:06:07.565799 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:07.565761    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:06:08.838228 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:08.838195    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"]
Sep 09 13:06:08.842275 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:08.842253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:08.844890 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:08.844862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c2e681-dockercfg-dwck7\""
Sep 09 13:06:08.845039 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:08.844905    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c2e681\""
Sep 09 13:06:08.850845 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:08.850792    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"]
Sep 09 13:06:08.920203 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:08.920159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djcbc\" (UniqueName: \"kubernetes.io/projected/f94906c2-df94-463a-a4ef-2b130ed30203-kube-api-access-djcbc\") pod \"payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h\" (UID: \"f94906c2-df94-463a-a4ef-2b130ed30203\") " pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:08.920203 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:08.920196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94906c2-df94-463a-a4ef-2b130ed30203-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h\" (UID: \"f94906c2-df94-463a-a4ef-2b130ed30203\") " pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:09.021518 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:09.021482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djcbc\" (UniqueName: \"kubernetes.io/projected/f94906c2-df94-463a-a4ef-2b130ed30203-kube-api-access-djcbc\") pod \"payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h\" (UID: \"f94906c2-df94-463a-a4ef-2b130ed30203\") " pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:09.021745 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:09.021529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94906c2-df94-463a-a4ef-2b130ed30203-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h\" (UID: \"f94906c2-df94-463a-a4ef-2b130ed30203\") " pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:09.022372 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:09.022295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94906c2-df94-463a-a4ef-2b130ed30203-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h\" (UID: \"f94906c2-df94-463a-a4ef-2b130ed30203\") " pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:09.030322 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:09.030301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djcbc\" (UniqueName: \"kubernetes.io/projected/f94906c2-df94-463a-a4ef-2b130ed30203-kube-api-access-djcbc\") pod \"payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h\" (UID: \"f94906c2-df94-463a-a4ef-2b130ed30203\") " pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:09.154753 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:09.154648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:09.428163 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:09.428137    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"]
Sep 09 13:06:09.430432 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:06:09.430404    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94906c2_df94_463a_a4ef_2b130ed30203.slice/crio-19542a91a12e179a68254e2a5bd71c2f6a74014cd6dfdba154a520e915c372c6 WatchSource:0}: Error finding container 19542a91a12e179a68254e2a5bd71c2f6a74014cd6dfdba154a520e915c372c6: Status 404 returned error can't find the container with id 19542a91a12e179a68254e2a5bd71c2f6a74014cd6dfdba154a520e915c372c6
Sep 09 13:06:09.567613 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:09.567579    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h" event={"ID":"f94906c2-df94-463a-a4ef-2b130ed30203","Type":"ContainerStarted","Data":"420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5"}
Sep 09 13:06:09.567795 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:09.567631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h" event={"ID":"f94906c2-df94-463a-a4ef-2b130ed30203","Type":"ContainerStarted","Data":"19542a91a12e179a68254e2a5bd71c2f6a74014cd6dfdba154a520e915c372c6"}
Sep 09 13:06:09.567795 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:09.567667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:09.583261 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:09.583194    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h" podStartSLOduration=1.5831717950000002 podStartE2EDuration="1.583171795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:06:09.581458154 +0000 UTC m=+1052.501349584" watchObservedRunningTime="2026-09-09 13:06:09.583171795 +0000 UTC m=+1052.503063138"
Sep 09 13:06:18.679255 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:18.679220    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"]
Sep 09 13:06:18.765564 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:18.765523    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:06:20.575736 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:20.575706    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:22.333958 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:22.333922    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"]
Sep 09 13:06:33.700591 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:33.700483    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2" podUID="9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" containerName="main" containerID="cri-o://8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103" gracePeriod=45
Sep 09 13:06:33.950555 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:33.950528    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:06:34.059385 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.059348    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-home\") pod \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") "
Sep 09 13:06:34.059582 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.059431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tls-certs\") pod \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") "
Sep 09 13:06:34.059582 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.059485    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hms72\" (UniqueName: \"kubernetes.io/projected/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-kube-api-access-hms72\") pod \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") "
Sep 09 13:06:34.059582 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.059513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-model-cache\") pod \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") "
Sep 09 13:06:34.059582 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.059558    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tmp-dir\") pod \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") "
Sep 09 13:06:34.059789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.059585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-dshm\") pod \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\" (UID: \"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2\") "
Sep 09 13:06:34.059789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.059608    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-home" (OuterVolumeSpecName: "home") pod "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" (UID: "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:34.059918 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.059852    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" (UID: "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:34.059918 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.059866    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" (UID: "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:34.060055 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.060037    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-model-cache\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:34.060094 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.060062    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tmp-dir\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:34.060094 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.060077    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-home\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:34.061910 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.061880    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-dshm" (OuterVolumeSpecName: "dshm") pod "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" (UID: "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:34.062031 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.061967    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" (UID: "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:06:34.062031 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.062000    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-kube-api-access-hms72" (OuterVolumeSpecName: "kube-api-access-hms72") pod "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" (UID: "9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2"). InnerVolumeSpecName "kube-api-access-hms72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:06:34.160713 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.160679    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-tls-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:34.160713 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.160708    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hms72\" (UniqueName: \"kubernetes.io/projected/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-kube-api-access-hms72\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:34.160713 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.160716    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2-dshm\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:34.711291 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.711259    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" containerID="8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103" exitCode=0
Sep 09 13:06:34.711715 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.711298    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2" event={"ID":"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2","Type":"ContainerDied","Data":"8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103"}
Sep 09 13:06:34.711715 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.711303    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"
Sep 09 13:06:34.711715 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.711330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2" event={"ID":"9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2","Type":"ContainerDied","Data":"40ffc9667a6fa7f19229afd115d760708debdfea0fd8d2b82935a79ac0f68344"}
Sep 09 13:06:34.711715 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.711347    2580 scope.go:117] "RemoveContainer" containerID="8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103"
Sep 09 13:06:34.720973 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.720957    2580 scope.go:117] "RemoveContainer" containerID="8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103"
Sep 09 13:06:34.721260 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:06:34.721240    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103\": container with ID starting with 8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103 not found: ID does not exist" containerID="8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103"
Sep 09 13:06:34.721348 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.721273    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103"} err="failed to get container status \"8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103\": rpc error: code = NotFound desc = could not find container \"8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103\": container with ID starting with 8fa033d5d51a76d17ad096c88f98551b11db52d8ba4f6abd267d0f7104bbb103 not found: ID does not exist"
Sep 09 13:06:34.732427 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.732403    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"]
Sep 09 13:06:34.735388 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.735367    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-a333cccb-kserve-7db8bb6796-j4kj2"]
Sep 09 13:06:34.968928 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:34.968839    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:06:35.593216 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:35.593187    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" path="/var/lib/kubelet/pods/9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2/volumes"
Sep 09 13:06:37.355598 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.355553    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh" podUID="7fdb3882-8344-4d47-a769-e7d20ee77a97" containerName="main" containerID="cri-o://b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e" gracePeriod=45
Sep 09 13:06:37.634069 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.634040    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:06:37.697663 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.697625    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-home\") pod \"7fdb3882-8344-4d47-a769-e7d20ee77a97\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") "
Sep 09 13:06:37.697890 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.697670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-model-cache\") pod \"7fdb3882-8344-4d47-a769-e7d20ee77a97\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") "
Sep 09 13:06:37.697890 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.697699    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fdb3882-8344-4d47-a769-e7d20ee77a97-tls-certs\") pod \"7fdb3882-8344-4d47-a769-e7d20ee77a97\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") "
Sep 09 13:06:37.697890 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.697770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-tmp-dir\") pod \"7fdb3882-8344-4d47-a769-e7d20ee77a97\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") "
Sep 09 13:06:37.697890 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.697855    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw45c\" (UniqueName: \"kubernetes.io/projected/7fdb3882-8344-4d47-a769-e7d20ee77a97-kube-api-access-lw45c\") pod \"7fdb3882-8344-4d47-a769-e7d20ee77a97\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") "
Sep 09 13:06:37.697890 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.697886    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-dshm\") pod \"7fdb3882-8344-4d47-a769-e7d20ee77a97\" (UID: \"7fdb3882-8344-4d47-a769-e7d20ee77a97\") "
Sep 09 13:06:37.698164 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.697985    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-home" (OuterVolumeSpecName: "home") pod "7fdb3882-8344-4d47-a769-e7d20ee77a97" (UID: "7fdb3882-8344-4d47-a769-e7d20ee77a97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:37.698164 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.698002    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fdb3882-8344-4d47-a769-e7d20ee77a97" (UID: "7fdb3882-8344-4d47-a769-e7d20ee77a97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:37.698164 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.698049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fdb3882-8344-4d47-a769-e7d20ee77a97" (UID: "7fdb3882-8344-4d47-a769-e7d20ee77a97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:37.698342 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.698317    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-home\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:37.698398 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.698343    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-model-cache\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:37.698398 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.698359    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-tmp-dir\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:37.700176 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.700149    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-dshm" (OuterVolumeSpecName: "dshm") pod "7fdb3882-8344-4d47-a769-e7d20ee77a97" (UID: "7fdb3882-8344-4d47-a769-e7d20ee77a97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:37.700284 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.700196    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fdb3882-8344-4d47-a769-e7d20ee77a97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fdb3882-8344-4d47-a769-e7d20ee77a97" (UID: "7fdb3882-8344-4d47-a769-e7d20ee77a97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:06:37.700284 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.700200    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdb3882-8344-4d47-a769-e7d20ee77a97-kube-api-access-lw45c" (OuterVolumeSpecName: "kube-api-access-lw45c") pod "7fdb3882-8344-4d47-a769-e7d20ee77a97" (UID: "7fdb3882-8344-4d47-a769-e7d20ee77a97"). InnerVolumeSpecName "kube-api-access-lw45c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:06:37.730729 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.730694    2580 generic.go:356] "Generic (PLEG): container finished" podID="7fdb3882-8344-4d47-a769-e7d20ee77a97" containerID="b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e" exitCode=0
Sep 09 13:06:37.730920 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.730748    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"
Sep 09 13:06:37.730920 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.730780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh" event={"ID":"7fdb3882-8344-4d47-a769-e7d20ee77a97","Type":"ContainerDied","Data":"b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e"}
Sep 09 13:06:37.730920 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.730853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh" event={"ID":"7fdb3882-8344-4d47-a769-e7d20ee77a97","Type":"ContainerDied","Data":"b88b93fbabf6df19fc27abe21501d02debbe202ab24ba75e53f708e7634fe341"}
Sep 09 13:06:37.730920 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.730875    2580 scope.go:117] "RemoveContainer" containerID="b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e"
Sep 09 13:06:37.740970 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.740952    2580 scope.go:117] "RemoveContainer" containerID="b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e"
Sep 09 13:06:37.741272 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:06:37.741235    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e\": container with ID starting with b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e not found: ID does not exist" containerID="b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e"
Sep 09 13:06:37.741349 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.741281    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e"} err="failed to get container status \"b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e\": rpc error: code = NotFound desc = could not find container \"b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e\": container with ID starting with b96dbddcbf40986c6d300f4f7c54e2b2d4854082310e1b5b868f51cabcf5d70e not found: ID does not exist"
Sep 09 13:06:37.751996 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.751972    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"]
Sep 09 13:06:37.753879 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.753860    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-a333cccb-kserve-55d9df8c4-r2shh"]
Sep 09 13:06:37.799530 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.799495    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw45c\" (UniqueName: \"kubernetes.io/projected/7fdb3882-8344-4d47-a769-e7d20ee77a97-kube-api-access-lw45c\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:37.799530 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.799524    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fdb3882-8344-4d47-a769-e7d20ee77a97-dshm\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:37.799530 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:37.799536    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fdb3882-8344-4d47-a769-e7d20ee77a97-tls-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:39.592053 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:39.592019    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fdb3882-8344-4d47-a769-e7d20ee77a97" path="/var/lib/kubelet/pods/7fdb3882-8344-4d47-a769-e7d20ee77a97/volumes"
Sep 09 13:06:40.850784 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.850737    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"]
Sep 09 13:06:40.851352 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.851330    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" containerName="main"
Sep 09 13:06:40.851450 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.851356    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" containerName="main"
Sep 09 13:06:40.851450 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.851375    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fdb3882-8344-4d47-a769-e7d20ee77a97" containerName="main"
Sep 09 13:06:40.851450 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.851383    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fdb3882-8344-4d47-a769-e7d20ee77a97" containerName="main"
Sep 09 13:06:40.851596 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.851526    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fdb3882-8344-4d47-a769-e7d20ee77a97" containerName="main"
Sep 09 13:06:40.851596 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.851542    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7bd5eb-98b5-4606-9da2-66c6bfa91ef2" containerName="main"
Sep 09 13:06:40.855916 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.855892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:40.858232 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.858212    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c2e681\"/\"kube-root-ca.crt\""
Sep 09 13:06:40.859392 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.859364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c2e681\"/\"default-dockercfg-zxph9\""
Sep 09 13:06:40.859627 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.859605    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c2e681\"/\"openshift-service-ca.crt\""
Sep 09 13:06:40.859723 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.859642    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c2e681\"/\"explicit-model-c2e681-kserve-self-signed-certs\""
Sep 09 13:06:40.862223 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.862201    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"]
Sep 09 13:06:40.932609 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.932575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-model-cache\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:40.932609 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.932614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88681aa4-6aed-40bd-98a0-a7d0abdea348-tls-certs\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:40.932870 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.932711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-tmp-dir\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:40.932870 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.932749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-home\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:40.932990 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.932868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-dshm\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:40.932990 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:40.932941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jbxj\" (UniqueName: \"kubernetes.io/projected/88681aa4-6aed-40bd-98a0-a7d0abdea348-kube-api-access-5jbxj\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.033671 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.033627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-dshm\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.033895 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.033688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jbxj\" (UniqueName: \"kubernetes.io/projected/88681aa4-6aed-40bd-98a0-a7d0abdea348-kube-api-access-5jbxj\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.033895 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.033715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-model-cache\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.033895 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.033734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88681aa4-6aed-40bd-98a0-a7d0abdea348-tls-certs\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.033895 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.033787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-tmp-dir\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.033895 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.033852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-home\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.034219 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.034185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-model-cache\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.034219 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.034196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-home\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.034434 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.034249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-tmp-dir\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.036196 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.036171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-dshm\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.036443 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.036426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88681aa4-6aed-40bd-98a0-a7d0abdea348-tls-certs\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.043649 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.043628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jbxj\" (UniqueName: \"kubernetes.io/projected/88681aa4-6aed-40bd-98a0-a7d0abdea348-kube-api-access-5jbxj\") pod \"explicit-model-c2e681-kserve-6b6c47bc47-b8gkq\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") " pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.166902 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.166794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:41.440322 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.440293    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"]
Sep 09 13:06:41.442399 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:06:41.442369    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88681aa4_6aed_40bd_98a0_a7d0abdea348.slice/crio-54df338a9322e8fba44e41ad3ba4b8528aeca623a54e08399b4448c6dee110af WatchSource:0}: Error finding container 54df338a9322e8fba44e41ad3ba4b8528aeca623a54e08399b4448c6dee110af: Status 404 returned error can't find the container with id 54df338a9322e8fba44e41ad3ba4b8528aeca623a54e08399b4448c6dee110af
Sep 09 13:06:41.757016 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.756919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq" event={"ID":"88681aa4-6aed-40bd-98a0-a7d0abdea348","Type":"ContainerStarted","Data":"41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf"}
Sep 09 13:06:41.757016 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.756963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq" event={"ID":"88681aa4-6aed-40bd-98a0-a7d0abdea348","Type":"ContainerStarted","Data":"54df338a9322e8fba44e41ad3ba4b8528aeca623a54e08399b4448c6dee110af"}
Sep 09 13:06:41.775801 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:41.775723    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq" podStartSLOduration=1.775700595 podStartE2EDuration="1.775700595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:06:41.772806554 +0000 UTC m=+1084.692697916" watchObservedRunningTime="2026-09-09 13:06:41.775700595 +0000 UTC m=+1084.695591933"
Sep 09 13:06:44.160956 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:44.160921    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"]
Sep 09 13:06:44.294735 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:44.294695    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"]
Sep 09 13:06:44.298140 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:44.298122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"
Sep 09 13:06:44.300599 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:44.300579    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:06:44.300918 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:44.300902    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t5rn5\""
Sep 09 13:06:44.304990 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:44.304966    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"]
Sep 09 13:06:44.473458 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:44.473362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e872a8-acd7-4f57-9b7e-549c44401382-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c2e681-26hf2\" (UID: \"19e872a8-acd7-4f57-9b7e-549c44401382\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"
Sep 09 13:06:44.574425 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:44.574379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e872a8-acd7-4f57-9b7e-549c44401382-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c2e681-26hf2\" (UID: \"19e872a8-acd7-4f57-9b7e-549c44401382\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"
Sep 09 13:06:44.575045 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:44.575026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e872a8-acd7-4f57-9b7e-549c44401382-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c2e681-26hf2\" (UID: \"19e872a8-acd7-4f57-9b7e-549c44401382\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"
Sep 09 13:06:44.611432 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:44.611397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"
Sep 09 13:06:44.887528 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:44.887504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"]
Sep 09 13:06:44.889859 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:06:44.889812    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e872a8_acd7_4f57_9b7e_549c44401382.slice/crio-02739617a111d8d33ccc099303fe146c5c585d3eb200889af48dc96bbd8068fb WatchSource:0}: Error finding container 02739617a111d8d33ccc099303fe146c5c585d3eb200889af48dc96bbd8068fb: Status 404 returned error can't find the container with id 02739617a111d8d33ccc099303fe146c5c585d3eb200889af48dc96bbd8068fb
Sep 09 13:06:45.778946 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:45.778905    2580 generic.go:356] "Generic (PLEG): container finished" podID="19e872a8-acd7-4f57-9b7e-549c44401382" containerID="7349ad750ff7784e8484601e661624075d5f28c6249fbde0af95f1619c6b4b14" exitCode=0
Sep 09 13:06:45.779315 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:45.778998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2" event={"ID":"19e872a8-acd7-4f57-9b7e-549c44401382","Type":"ContainerDied","Data":"7349ad750ff7784e8484601e661624075d5f28c6249fbde0af95f1619c6b4b14"}
Sep 09 13:06:45.779315 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:45.779061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2" event={"ID":"19e872a8-acd7-4f57-9b7e-549c44401382","Type":"ContainerStarted","Data":"02739617a111d8d33ccc099303fe146c5c585d3eb200889af48dc96bbd8068fb"}
Sep 09 13:06:46.063855 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:46.063802    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:06:46.920777 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:46.920753    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"
Sep 09 13:06:47.100071 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:47.100041    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e872a8-acd7-4f57-9b7e-549c44401382-maas-api-service-ca\") pod \"19e872a8-acd7-4f57-9b7e-549c44401382\" (UID: \"19e872a8-acd7-4f57-9b7e-549c44401382\") "
Sep 09 13:06:47.100540 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:47.100511    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e872a8-acd7-4f57-9b7e-549c44401382-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19e872a8-acd7-4f57-9b7e-549c44401382" (UID: "19e872a8-acd7-4f57-9b7e-549c44401382"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:06:47.201255 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:47.201220    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e872a8-acd7-4f57-9b7e-549c44401382-maas-api-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:47.791029 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:47.791000    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"
Sep 09 13:06:47.791029 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:47.791019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2" event={"ID":"19e872a8-acd7-4f57-9b7e-549c44401382","Type":"ContainerDied","Data":"02739617a111d8d33ccc099303fe146c5c585d3eb200889af48dc96bbd8068fb"}
Sep 09 13:06:47.791243 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:47.791067    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02739617a111d8d33ccc099303fe146c5c585d3eb200889af48dc96bbd8068fb"
Sep 09 13:06:51.167409 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:51.167362    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:55.037135 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.037099    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"]
Sep 09 13:06:55.037564 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.037316    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h" podUID="f94906c2-df94-463a-a4ef-2b130ed30203" containerName="payload-processing" containerID="cri-o://420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5" gracePeriod=30
Sep 09 13:06:55.266600 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.266531    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"]
Sep 09 13:06:55.269489 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.269459    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c2e681-26hf2"]
Sep 09 13:06:55.305215 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.305176    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:55.374865 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.374809    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djcbc\" (UniqueName: \"kubernetes.io/projected/f94906c2-df94-463a-a4ef-2b130ed30203-kube-api-access-djcbc\") pod \"f94906c2-df94-463a-a4ef-2b130ed30203\" (UID: \"f94906c2-df94-463a-a4ef-2b130ed30203\") "
Sep 09 13:06:55.375050 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.374985    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94906c2-df94-463a-a4ef-2b130ed30203-plugins-config-volume\") pod \"f94906c2-df94-463a-a4ef-2b130ed30203\" (UID: \"f94906c2-df94-463a-a4ef-2b130ed30203\") "
Sep 09 13:06:55.375350 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.375325    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94906c2-df94-463a-a4ef-2b130ed30203-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f94906c2-df94-463a-a4ef-2b130ed30203" (UID: "f94906c2-df94-463a-a4ef-2b130ed30203"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:06:55.377165 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.377139    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94906c2-df94-463a-a4ef-2b130ed30203-kube-api-access-djcbc" (OuterVolumeSpecName: "kube-api-access-djcbc") pod "f94906c2-df94-463a-a4ef-2b130ed30203" (UID: "f94906c2-df94-463a-a4ef-2b130ed30203"). InnerVolumeSpecName "kube-api-access-djcbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:06:55.475794 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.475702    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94906c2-df94-463a-a4ef-2b130ed30203-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:55.475794 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.475730    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djcbc\" (UniqueName: \"kubernetes.io/projected/f94906c2-df94-463a-a4ef-2b130ed30203-kube-api-access-djcbc\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:55.592019 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.591984    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19e872a8-acd7-4f57-9b7e-549c44401382" path="/var/lib/kubelet/pods/19e872a8-acd7-4f57-9b7e-549c44401382/volumes"
Sep 09 13:06:55.834282 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.834249    2580 generic.go:356] "Generic (PLEG): container finished" podID="f94906c2-df94-463a-a4ef-2b130ed30203" containerID="420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5" exitCode=0
Sep 09 13:06:55.834498 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.834297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h" event={"ID":"f94906c2-df94-463a-a4ef-2b130ed30203","Type":"ContainerDied","Data":"420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5"}
Sep 09 13:06:55.834498 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.834304    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"
Sep 09 13:06:55.834498 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.834340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h" event={"ID":"f94906c2-df94-463a-a4ef-2b130ed30203","Type":"ContainerDied","Data":"19542a91a12e179a68254e2a5bd71c2f6a74014cd6dfdba154a520e915c372c6"}
Sep 09 13:06:55.834498 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.834363    2580 scope.go:117] "RemoveContainer" containerID="420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5"
Sep 09 13:06:55.843465 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.843444    2580 scope.go:117] "RemoveContainer" containerID="420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5"
Sep 09 13:06:55.843747 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:06:55.843725    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5\": container with ID starting with 420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5 not found: ID does not exist" containerID="420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5"
Sep 09 13:06:55.843799 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.843751    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5"} err="failed to get container status \"420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5\": rpc error: code = NotFound desc = could not find container \"420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5\": container with ID starting with 420f72b1e7855517becd0432f669c81476f814fc1ae97de3c02bbbb7adb3d2a5 not found: ID does not exist"
Sep 09 13:06:55.851207 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.851182    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"]
Sep 09 13:06:55.856326 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:55.856306    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c2e681-67d8d488b5-dzb9h"]
Sep 09 13:06:57.592134 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:57.592089    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f94906c2-df94-463a-a4ef-2b130ed30203" path="/var/lib/kubelet/pods/f94906c2-df94-463a-a4ef-2b130ed30203/volumes"
Sep 09 13:06:59.187038 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.186995    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq" podUID="88681aa4-6aed-40bd-98a0-a7d0abdea348" containerName="main" containerID="cri-o://41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf" gracePeriod=45
Sep 09 13:06:59.446374 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.446308    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:59.514961 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.514930    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-model-cache\") pod \"88681aa4-6aed-40bd-98a0-a7d0abdea348\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") "
Sep 09 13:06:59.515160 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.514976    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-tmp-dir\") pod \"88681aa4-6aed-40bd-98a0-a7d0abdea348\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") "
Sep 09 13:06:59.515160 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.514993    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-home\") pod \"88681aa4-6aed-40bd-98a0-a7d0abdea348\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") "
Sep 09 13:06:59.515160 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.515019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jbxj\" (UniqueName: \"kubernetes.io/projected/88681aa4-6aed-40bd-98a0-a7d0abdea348-kube-api-access-5jbxj\") pod \"88681aa4-6aed-40bd-98a0-a7d0abdea348\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") "
Sep 09 13:06:59.515160 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.515042    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88681aa4-6aed-40bd-98a0-a7d0abdea348-tls-certs\") pod \"88681aa4-6aed-40bd-98a0-a7d0abdea348\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") "
Sep 09 13:06:59.515160 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.515101    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-dshm\") pod \"88681aa4-6aed-40bd-98a0-a7d0abdea348\" (UID: \"88681aa4-6aed-40bd-98a0-a7d0abdea348\") "
Sep 09 13:06:59.515407 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.515254    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-model-cache" (OuterVolumeSpecName: "model-cache") pod "88681aa4-6aed-40bd-98a0-a7d0abdea348" (UID: "88681aa4-6aed-40bd-98a0-a7d0abdea348"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:59.515407 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.515263    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88681aa4-6aed-40bd-98a0-a7d0abdea348" (UID: "88681aa4-6aed-40bd-98a0-a7d0abdea348"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:59.515407 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.515304    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-home" (OuterVolumeSpecName: "home") pod "88681aa4-6aed-40bd-98a0-a7d0abdea348" (UID: "88681aa4-6aed-40bd-98a0-a7d0abdea348"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:59.515573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.515514    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-model-cache\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:59.515573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.515546    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-tmp-dir\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:59.515573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.515555    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-home\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:59.517492 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.517458    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88681aa4-6aed-40bd-98a0-a7d0abdea348-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88681aa4-6aed-40bd-98a0-a7d0abdea348" (UID: "88681aa4-6aed-40bd-98a0-a7d0abdea348"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:06:59.517492 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.517481    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-dshm" (OuterVolumeSpecName: "dshm") pod "88681aa4-6aed-40bd-98a0-a7d0abdea348" (UID: "88681aa4-6aed-40bd-98a0-a7d0abdea348"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:06:59.517492 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.517487    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88681aa4-6aed-40bd-98a0-a7d0abdea348-kube-api-access-5jbxj" (OuterVolumeSpecName: "kube-api-access-5jbxj") pod "88681aa4-6aed-40bd-98a0-a7d0abdea348" (UID: "88681aa4-6aed-40bd-98a0-a7d0abdea348"). InnerVolumeSpecName "kube-api-access-5jbxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:06:59.616629 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.616593    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jbxj\" (UniqueName: \"kubernetes.io/projected/88681aa4-6aed-40bd-98a0-a7d0abdea348-kube-api-access-5jbxj\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:59.616629 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.616619    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88681aa4-6aed-40bd-98a0-a7d0abdea348-tls-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:59.616629 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.616628    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88681aa4-6aed-40bd-98a0-a7d0abdea348-dshm\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:06:59.858273 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.858233    2580 generic.go:356] "Generic (PLEG): container finished" podID="88681aa4-6aed-40bd-98a0-a7d0abdea348" containerID="41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf" exitCode=0
Sep 09 13:06:59.858467 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.858278    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"
Sep 09 13:06:59.858467 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.858318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq" event={"ID":"88681aa4-6aed-40bd-98a0-a7d0abdea348","Type":"ContainerDied","Data":"41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf"}
Sep 09 13:06:59.858467 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.858368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq" event={"ID":"88681aa4-6aed-40bd-98a0-a7d0abdea348","Type":"ContainerDied","Data":"54df338a9322e8fba44e41ad3ba4b8528aeca623a54e08399b4448c6dee110af"}
Sep 09 13:06:59.858467 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.858385    2580 scope.go:117] "RemoveContainer" containerID="41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf"
Sep 09 13:06:59.868002 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.867977    2580 scope.go:117] "RemoveContainer" containerID="41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf"
Sep 09 13:06:59.868347 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:06:59.868325    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf\": container with ID starting with 41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf not found: ID does not exist" containerID="41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf"
Sep 09 13:06:59.868426 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.868357    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf"} err="failed to get container status \"41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf\": rpc error: code = NotFound desc = could not find container \"41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf\": container with ID starting with 41484864440edc18fbb71a5f28e0f87feddfdddcf88924e2a2fb066ba9e9fcbf not found: ID does not exist"
Sep 09 13:06:59.874216 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.874182    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"]
Sep 09 13:06:59.879134 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:06:59.879097    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c2e681/explicit-model-c2e681-kserve-6b6c47bc47-b8gkq"]
Sep 09 13:07:01.591967 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:01.591932    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88681aa4-6aed-40bd-98a0-a7d0abdea348" path="/var/lib/kubelet/pods/88681aa4-6aed-40bd-98a0-a7d0abdea348/volumes"
Sep 09 13:07:02.854282 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.854248    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"]
Sep 09 13:07:02.854925 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.854908    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e872a8-acd7-4f57-9b7e-549c44401382" containerName="revoke-keys"
Sep 09 13:07:02.854992 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.854929    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e872a8-acd7-4f57-9b7e-549c44401382" containerName="revoke-keys"
Sep 09 13:07:02.854992 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.854956    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88681aa4-6aed-40bd-98a0-a7d0abdea348" containerName="main"
Sep 09 13:07:02.854992 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.854964    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88681aa4-6aed-40bd-98a0-a7d0abdea348" containerName="main"
Sep 09 13:07:02.854992 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.854989    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f94906c2-df94-463a-a4ef-2b130ed30203" containerName="payload-processing"
Sep 09 13:07:02.855116 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.854997    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f94906c2-df94-463a-a4ef-2b130ed30203" containerName="payload-processing"
Sep 09 13:07:02.855116 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.855093    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f94906c2-df94-463a-a4ef-2b130ed30203" containerName="payload-processing"
Sep 09 13:07:02.855116 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.855104    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e872a8-acd7-4f57-9b7e-549c44401382" containerName="revoke-keys"
Sep 09 13:07:02.855116 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.855110    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="88681aa4-6aed-40bd-98a0-a7d0abdea348" containerName="main"
Sep 09 13:07:02.859721 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.859698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:02.862261 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.862233    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-11f768-openshift-default-dockercfg-27hfb\""
Sep 09 13:07:02.868750 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.868725    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"]
Sep 09 13:07:02.950008 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.949972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-socket\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:02.950221 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.950017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-certs\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:02.950221 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.950039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-envoy\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:02.950221 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.950133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwzhh\" (UniqueName: \"kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-kube-api-access-qwzhh\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:02.950221 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.950159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-credential-socket\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:02.950221 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.950211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-data\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:02.950440 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.950244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-istio-token\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:02.950440 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.950278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11617c63-0e15-4583-97f4-e459e7b8a413-istiod-ca-cert\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:02.950440 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:02.950340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11617c63-0e15-4583-97f4-e459e7b8a413-istio-podinfo\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.051749 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.051706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-socket\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.051966 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.051762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-certs\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.051966 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.051790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-envoy\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.051966 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.051894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwzhh\" (UniqueName: \"kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-kube-api-access-qwzhh\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.051966 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.051923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-credential-socket\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.052199 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.051979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-data\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.052199 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.052009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-istio-token\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.052199 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.052037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11617c63-0e15-4583-97f4-e459e7b8a413-istiod-ca-cert\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.052199 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.052064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11617c63-0e15-4583-97f4-e459e7b8a413-istio-podinfo\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.052199 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.052165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-socket\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.052199 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.052187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-certs\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.052523 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.052448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-data\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.052585 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.052531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-credential-socket\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.052955 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.052909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11617c63-0e15-4583-97f4-e459e7b8a413-istiod-ca-cert\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.054357 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.054335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-envoy\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.054656 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.054637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11617c63-0e15-4583-97f4-e459e7b8a413-istio-podinfo\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.059725 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.059699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-istio-token\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.060112 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.060090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwzhh\" (UniqueName: \"kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-kube-api-access-qwzhh\") pod \"e2e-noten-11f768-openshift-default-7d896848bb-chp8c\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") " pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.172132 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.172035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:03.371139 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.371101    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:07:03.448209 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.448186    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"]
Sep 09 13:07:03.450703 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:07:03.450678    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11617c63_0e15_4583_97f4_e459e7b8a413.slice/crio-c09d3c9c2b60b75a3dee8b5fd1bf3e65ca819b7352ca313373085ebe78c5dc21 WatchSource:0}: Error finding container c09d3c9c2b60b75a3dee8b5fd1bf3e65ca819b7352ca313373085ebe78c5dc21: Status 404 returned error can't find the container with id c09d3c9c2b60b75a3dee8b5fd1bf3e65ca819b7352ca313373085ebe78c5dc21
Sep 09 13:07:03.453113 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.453081    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:07:03.453193 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.453151    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:07:03.453193 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.453178    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:07:03.882429 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.882395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c" event={"ID":"11617c63-0e15-4583-97f4-e459e7b8a413","Type":"ContainerStarted","Data":"f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd"}
Sep 09 13:07:03.882878 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.882444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c" event={"ID":"11617c63-0e15-4583-97f4-e459e7b8a413","Type":"ContainerStarted","Data":"c09d3c9c2b60b75a3dee8b5fd1bf3e65ca819b7352ca313373085ebe78c5dc21"}
Sep 09 13:07:03.901954 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:03.901900    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c" podStartSLOduration=1.901882771 podStartE2EDuration="1.901882771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:07:03.900077944 +0000 UTC m=+1106.819969285" watchObservedRunningTime="2026-09-09 13:07:03.901882771 +0000 UTC m=+1106.821774111"
Sep 09 13:07:04.173053 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:04.172964    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:04.178456 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:04.178432    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:04.888376 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:04.888346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:04.889400 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:04.889382    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:16.571614 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.571580    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"]
Sep 09 13:07:16.575676 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.575656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.578508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.578483    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-73011135-serving-cert\""
Sep 09 13:07:16.579600 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.579578    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 13:07:16.579740 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.579579    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 13:07:16.579740 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.579635    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:07:16.579740 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.579585    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dbjk4\""
Sep 09 13:07:16.587289 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.587263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"]
Sep 09 13:07:16.694684 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.694643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-maas-api-tls\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.694913 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.694749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-metrics-certs\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.694913 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.694850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q8tb\" (UniqueName: \"kubernetes.io/projected/2e319adc-3828-4aea-8078-c5c28a632053-kube-api-access-9q8tb\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.695043 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.694939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e319adc-3828-4aea-8078-c5c28a632053-openshift-service-ca\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.795685 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.795645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q8tb\" (UniqueName: \"kubernetes.io/projected/2e319adc-3828-4aea-8078-c5c28a632053-kube-api-access-9q8tb\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.795926 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.795704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e319adc-3828-4aea-8078-c5c28a632053-openshift-service-ca\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.795926 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.795765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-maas-api-tls\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.795926 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.795805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-metrics-certs\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.796464 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.796438    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e319adc-3828-4aea-8078-c5c28a632053-openshift-service-ca\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.798563 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.798536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-metrics-certs\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.798663 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.798583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-maas-api-tls\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.803703 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.803680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q8tb\" (UniqueName: \"kubernetes.io/projected/2e319adc-3828-4aea-8078-c5c28a632053-kube-api-access-9q8tb\") pod \"maas-api-e2e-mt-73011135-ff74bfb59-wg5m9\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:16.888744 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:16.888643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:17.165871 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:17.165846    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"]
Sep 09 13:07:17.167989 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:07:17.167954    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e319adc_3828_4aea_8078_c5c28a632053.slice/crio-bb4f8186d2e553a86191a7ffaade9117627e021890f84ad08b4fb1f4f7e8b77f WatchSource:0}: Error finding container bb4f8186d2e553a86191a7ffaade9117627e021890f84ad08b4fb1f4f7e8b77f: Status 404 returned error can't find the container with id bb4f8186d2e553a86191a7ffaade9117627e021890f84ad08b4fb1f4f7e8b77f
Sep 09 13:07:17.661637 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:17.661603    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:07:17.964464 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:17.964377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9" event={"ID":"2e319adc-3828-4aea-8078-c5c28a632053","Type":"ContainerStarted","Data":"09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717"}
Sep 09 13:07:17.964464 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:17.964435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9" event={"ID":"2e319adc-3828-4aea-8078-c5c28a632053","Type":"ContainerStarted","Data":"bb4f8186d2e553a86191a7ffaade9117627e021890f84ad08b4fb1f4f7e8b77f"}
Sep 09 13:07:17.964649 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:17.964507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:17.981132 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:17.981084    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9" podStartSLOduration=1.5688098259999999 podStartE2EDuration="1.981068675s" totalImagesPullingTime="412.258849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:07:17.979644353 +0000 UTC m=+1120.899535693" watchObservedRunningTime="2026-09-09 13:07:17.981068675 +0000 UTC m=+1120.900960014"
Sep 09 13:07:22.017982 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.017944    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"]
Sep 09 13:07:22.021831 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.021803    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.024348 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.024325    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-11f768\"/\"kube-root-ca.crt\""
Sep 09 13:07:22.025491 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.025469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-11f768\"/\"default-dockercfg-tkv4f\""
Sep 09 13:07:22.025605 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.025509    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-11f768\"/\"noten-model-11f768-kserve-self-signed-certs\""
Sep 09 13:07:22.025605 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.025546    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-11f768\"/\"openshift-service-ca.crt\""
Sep 09 13:07:22.029372 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.029346    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"]
Sep 09 13:07:22.154657 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.154618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-dshm\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.154851 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.154666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-home\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.154851 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.154762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322ef63c-182b-4ce2-aea7-8d666b625c19-tls-certs\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.154851 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.154788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8njsv\" (UniqueName: \"kubernetes.io/projected/322ef63c-182b-4ce2-aea7-8d666b625c19-kube-api-access-8njsv\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.154980 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.154879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-model-cache\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.154980 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.154922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-tmp-dir\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.255503 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.255462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-model-cache\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.255676 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.255536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-tmp-dir\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.255676 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.255596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-dshm\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.255676 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.255634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-home\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.255887 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.255694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322ef63c-182b-4ce2-aea7-8d666b625c19-tls-certs\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.255887 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.255720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8njsv\" (UniqueName: \"kubernetes.io/projected/322ef63c-182b-4ce2-aea7-8d666b625c19-kube-api-access-8njsv\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.255991 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.255889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-model-cache\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.255991 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.255957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-tmp-dir\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.256082 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.256044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-home\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.258080 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.258051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-dshm\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.258338 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.258319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322ef63c-182b-4ce2-aea7-8d666b625c19-tls-certs\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.263675 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.263656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8njsv\" (UniqueName: \"kubernetes.io/projected/322ef63c-182b-4ce2-aea7-8d666b625c19-kube-api-access-8njsv\") pod \"noten-model-11f768-kserve-6fd64b98dc-fbxd9\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") " pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.332703 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.332611    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:22.607186 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.607099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"]
Sep 09 13:07:22.610294 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:07:22.610260    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod322ef63c_182b_4ce2_aea7_8d666b625c19.slice/crio-595d2e4a074346d8ecf06cd789373942e13ed5b4d76031fd0b65a414747faa9e WatchSource:0}: Error finding container 595d2e4a074346d8ecf06cd789373942e13ed5b4d76031fd0b65a414747faa9e: Status 404 returned error can't find the container with id 595d2e4a074346d8ecf06cd789373942e13ed5b4d76031fd0b65a414747faa9e
Sep 09 13:07:22.992252 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.992159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9" event={"ID":"322ef63c-182b-4ce2-aea7-8d666b625c19","Type":"ContainerStarted","Data":"2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22"}
Sep 09 13:07:22.992252 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:22.992207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9" event={"ID":"322ef63c-182b-4ce2-aea7-8d666b625c19","Type":"ContainerStarted","Data":"595d2e4a074346d8ecf06cd789373942e13ed5b4d76031fd0b65a414747faa9e"}
Sep 09 13:07:23.009584 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:23.009538    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9" podStartSLOduration=2.009521006 podStartE2EDuration="2.009521006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:07:23.007957535 +0000 UTC m=+1125.927848886" watchObservedRunningTime="2026-09-09 13:07:23.009521006 +0000 UTC m=+1125.929412344"
Sep 09 13:07:23.975886 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:23.975845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:26.154264 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:26.154222    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"]
Sep 09 13:07:26.285088 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:26.285036    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"]
Sep 09 13:07:26.285458 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:26.285398    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c" podUID="11617c63-0e15-4583-97f4-e459e7b8a413" containerName="istio-proxy" containerID="cri-o://f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd" gracePeriod=30
Sep 09 13:07:30.694277 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:30.694241    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"]
Sep 09 13:07:30.698081 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:30.698063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"
Sep 09 13:07:30.700575 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:30.700552    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t5rn5\""
Sep 09 13:07:30.704566 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:30.704540    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"]
Sep 09 13:07:30.741941 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:30.741914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa94e944-c387-4939-993a-49a62555d5be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6\" (UID: \"fa94e944-c387-4939-993a-49a62555d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"
Sep 09 13:07:30.843229 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:30.843194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa94e944-c387-4939-993a-49a62555d5be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6\" (UID: \"fa94e944-c387-4939-993a-49a62555d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"
Sep 09 13:07:30.843936 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:30.843915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa94e944-c387-4939-993a-49a62555d5be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6\" (UID: \"fa94e944-c387-4939-993a-49a62555d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"
Sep 09 13:07:31.011509 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.011406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"
Sep 09 13:07:31.284652 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.284628    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"]
Sep 09 13:07:31.286654 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:07:31.286627    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa94e944_c387_4939_993a_49a62555d5be.slice/crio-ca07aaa0202816dbe3da6e7a51cfe01933ffcd2faab203e3c16b3ba961cc3d8b WatchSource:0}: Error finding container ca07aaa0202816dbe3da6e7a51cfe01933ffcd2faab203e3c16b3ba961cc3d8b: Status 404 returned error can't find the container with id ca07aaa0202816dbe3da6e7a51cfe01933ffcd2faab203e3c16b3ba961cc3d8b
Sep 09 13:07:31.533680 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.533622    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:31.653402 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653368    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11617c63-0e15-4583-97f4-e459e7b8a413-istio-podinfo\") pod \"11617c63-0e15-4583-97f4-e459e7b8a413\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") "
Sep 09 13:07:31.653595 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653426    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-istio-token\") pod \"11617c63-0e15-4583-97f4-e459e7b8a413\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") "
Sep 09 13:07:31.653595 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11617c63-0e15-4583-97f4-e459e7b8a413-istiod-ca-cert\") pod \"11617c63-0e15-4583-97f4-e459e7b8a413\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") "
Sep 09 13:07:31.653595 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-data\") pod \"11617c63-0e15-4583-97f4-e459e7b8a413\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") "
Sep 09 13:07:31.653595 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653517    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-credential-socket\") pod \"11617c63-0e15-4583-97f4-e459e7b8a413\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") "
Sep 09 13:07:31.653595 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-certs\") pod \"11617c63-0e15-4583-97f4-e459e7b8a413\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") "
Sep 09 13:07:31.653595 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653577    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-socket\") pod \"11617c63-0e15-4583-97f4-e459e7b8a413\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") "
Sep 09 13:07:31.653941 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-envoy\") pod \"11617c63-0e15-4583-97f4-e459e7b8a413\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") "
Sep 09 13:07:31.653941 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653678    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwzhh\" (UniqueName: \"kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-kube-api-access-qwzhh\") pod \"11617c63-0e15-4583-97f4-e459e7b8a413\" (UID: \"11617c63-0e15-4583-97f4-e459e7b8a413\") "
Sep 09 13:07:31.653941 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653852    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11617c63-0e15-4583-97f4-e459e7b8a413" (UID: "11617c63-0e15-4583-97f4-e459e7b8a413"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:07:31.653941 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-data" (OuterVolumeSpecName: "istio-data") pod "11617c63-0e15-4583-97f4-e459e7b8a413" (UID: "11617c63-0e15-4583-97f4-e459e7b8a413"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:07:31.654149 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.653935    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11617c63-0e15-4583-97f4-e459e7b8a413-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11617c63-0e15-4583-97f4-e459e7b8a413" (UID: "11617c63-0e15-4583-97f4-e459e7b8a413"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:07:31.654149 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.654095    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11617c63-0e15-4583-97f4-e459e7b8a413" (UID: "11617c63-0e15-4583-97f4-e459e7b8a413"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:07:31.654276 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.654158    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11617c63-0e15-4583-97f4-e459e7b8a413" (UID: "11617c63-0e15-4583-97f4-e459e7b8a413"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:07:31.654587 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.654534    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11617c63-0e15-4583-97f4-e459e7b8a413-istiod-ca-cert\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:31.654587 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.654563    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-data\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:31.654587 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.654578    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-credential-socket\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:31.654587 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.654591    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:31.654930 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.654606    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-workload-socket\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:31.655978 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.655957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11617c63-0e15-4583-97f4-e459e7b8a413-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11617c63-0e15-4583-97f4-e459e7b8a413" (UID: "11617c63-0e15-4583-97f4-e459e7b8a413"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:07:31.655978 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.655962    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-istio-token" (OuterVolumeSpecName: "istio-token") pod "11617c63-0e15-4583-97f4-e459e7b8a413" (UID: "11617c63-0e15-4583-97f4-e459e7b8a413"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:07:31.656135 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.656001    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11617c63-0e15-4583-97f4-e459e7b8a413" (UID: "11617c63-0e15-4583-97f4-e459e7b8a413"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:07:31.656224 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.656208    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-kube-api-access-qwzhh" (OuterVolumeSpecName: "kube-api-access-qwzhh") pod "11617c63-0e15-4583-97f4-e459e7b8a413" (UID: "11617c63-0e15-4583-97f4-e459e7b8a413"). InnerVolumeSpecName "kube-api-access-qwzhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:07:31.755859 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.755804    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwzhh\" (UniqueName: \"kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-kube-api-access-qwzhh\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:31.755859 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.755859    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11617c63-0e15-4583-97f4-e459e7b8a413-istio-podinfo\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:31.756267 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.755871    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11617c63-0e15-4583-97f4-e459e7b8a413-istio-token\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:31.756267 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:31.755880    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11617c63-0e15-4583-97f4-e459e7b8a413-istio-envoy\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:32.047863 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.047800    2580 generic.go:356] "Generic (PLEG): container finished" podID="11617c63-0e15-4583-97f4-e459e7b8a413" containerID="f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd" exitCode=0
Sep 09 13:07:32.048035 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.047881    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"
Sep 09 13:07:32.048035 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.047890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c" event={"ID":"11617c63-0e15-4583-97f4-e459e7b8a413","Type":"ContainerDied","Data":"f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd"}
Sep 09 13:07:32.048035 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.047948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c" event={"ID":"11617c63-0e15-4583-97f4-e459e7b8a413","Type":"ContainerDied","Data":"c09d3c9c2b60b75a3dee8b5fd1bf3e65ca819b7352ca313373085ebe78c5dc21"}
Sep 09 13:07:32.048035 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.047977    2580 scope.go:117] "RemoveContainer" containerID="f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd"
Sep 09 13:07:32.049724 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.049697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6" event={"ID":"fa94e944-c387-4939-993a-49a62555d5be","Type":"ContainerStarted","Data":"d827dec0c20ef1e0eeae59493b5f49b69a5a00e1f5e8c8078873464c0646a582"}
Sep 09 13:07:32.049869 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.049746    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6" event={"ID":"fa94e944-c387-4939-993a-49a62555d5be","Type":"ContainerStarted","Data":"ca07aaa0202816dbe3da6e7a51cfe01933ffcd2faab203e3c16b3ba961cc3d8b"}
Sep 09 13:07:32.058413 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.058395    2580 scope.go:117] "RemoveContainer" containerID="f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd"
Sep 09 13:07:32.058708 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:07:32.058688    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd\": container with ID starting with f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd not found: ID does not exist" containerID="f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd"
Sep 09 13:07:32.058769 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.058720    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd"} err="failed to get container status \"f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd\": rpc error: code = NotFound desc = could not find container \"f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd\": container with ID starting with f7fee850f1842240d434b98f10c335c6acb93e5d650753be97254283062a86cd not found: ID does not exist"
Sep 09 13:07:32.066294 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.066248    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6" podStartSLOduration=2.066236369 podStartE2EDuration="2.066236369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:07:32.064650396 +0000 UTC m=+1134.984541714" watchObservedRunningTime="2026-09-09 13:07:32.066236369 +0000 UTC m=+1134.986127708"
Sep 09 13:07:32.077416 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.077388    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"]
Sep 09 13:07:32.081096 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.081075    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-11f768-openshift-default-7d896848bb-chp8c"]
Sep 09 13:07:32.333561 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:32.333527    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:33.057947 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:33.057907    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa94e944-c387-4939-993a-49a62555d5be" containerID="d827dec0c20ef1e0eeae59493b5f49b69a5a00e1f5e8c8078873464c0646a582" exitCode=0
Sep 09 13:07:33.058414 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:33.057988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6" event={"ID":"fa94e944-c387-4939-993a-49a62555d5be","Type":"ContainerDied","Data":"d827dec0c20ef1e0eeae59493b5f49b69a5a00e1f5e8c8078873464c0646a582"}
Sep 09 13:07:33.591956 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:33.591924    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11617c63-0e15-4583-97f4-e459e7b8a413" path="/var/lib/kubelet/pods/11617c63-0e15-4583-97f4-e459e7b8a413/volumes"
Sep 09 13:07:34.198954 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:34.198928    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"
Sep 09 13:07:34.281585 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:34.281544    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa94e944-c387-4939-993a-49a62555d5be-maas-api-service-ca\") pod \"fa94e944-c387-4939-993a-49a62555d5be\" (UID: \"fa94e944-c387-4939-993a-49a62555d5be\") "
Sep 09 13:07:34.281976 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:34.281951    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa94e944-c387-4939-993a-49a62555d5be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa94e944-c387-4939-993a-49a62555d5be" (UID: "fa94e944-c387-4939-993a-49a62555d5be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:07:34.383382 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:34.383297    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa94e944-c387-4939-993a-49a62555d5be-maas-api-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:35.070616 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:35.070586    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"
Sep 09 13:07:35.070616 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:35.070621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6" event={"ID":"fa94e944-c387-4939-993a-49a62555d5be","Type":"ContainerDied","Data":"ca07aaa0202816dbe3da6e7a51cfe01933ffcd2faab203e3c16b3ba961cc3d8b"}
Sep 09 13:07:35.070851 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:35.070665    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca07aaa0202816dbe3da6e7a51cfe01933ffcd2faab203e3c16b3ba961cc3d8b"
Sep 09 13:07:35.562858 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:35.562808    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:07:41.175257 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.175215    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9" podUID="322ef63c-182b-4ce2-aea7-8d666b625c19" containerName="main" containerID="cri-o://2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22" gracePeriod=45
Sep 09 13:07:41.431050 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.430970    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:41.557636 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.557596    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-dshm\") pod \"322ef63c-182b-4ce2-aea7-8d666b625c19\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") "
Sep 09 13:07:41.557875 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.557683    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-home\") pod \"322ef63c-182b-4ce2-aea7-8d666b625c19\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") "
Sep 09 13:07:41.557875 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.557711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8njsv\" (UniqueName: \"kubernetes.io/projected/322ef63c-182b-4ce2-aea7-8d666b625c19-kube-api-access-8njsv\") pod \"322ef63c-182b-4ce2-aea7-8d666b625c19\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") "
Sep 09 13:07:41.557875 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.557767    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322ef63c-182b-4ce2-aea7-8d666b625c19-tls-certs\") pod \"322ef63c-182b-4ce2-aea7-8d666b625c19\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") "
Sep 09 13:07:41.557875 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.557802    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-model-cache\") pod \"322ef63c-182b-4ce2-aea7-8d666b625c19\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") "
Sep 09 13:07:41.558104 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.557910    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-tmp-dir\") pod \"322ef63c-182b-4ce2-aea7-8d666b625c19\" (UID: \"322ef63c-182b-4ce2-aea7-8d666b625c19\") "
Sep 09 13:07:41.558104 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.557964    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-home" (OuterVolumeSpecName: "home") pod "322ef63c-182b-4ce2-aea7-8d666b625c19" (UID: "322ef63c-182b-4ce2-aea7-8d666b625c19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:07:41.558200 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.558141    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-model-cache" (OuterVolumeSpecName: "model-cache") pod "322ef63c-182b-4ce2-aea7-8d666b625c19" (UID: "322ef63c-182b-4ce2-aea7-8d666b625c19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:07:41.558256 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.558228    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "322ef63c-182b-4ce2-aea7-8d666b625c19" (UID: "322ef63c-182b-4ce2-aea7-8d666b625c19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:07:41.558421 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.558383    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-model-cache\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:41.558421 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.558407    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-tmp-dir\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:41.558421 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.558419    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-home\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:41.560147 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.560124    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322ef63c-182b-4ce2-aea7-8d666b625c19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "322ef63c-182b-4ce2-aea7-8d666b625c19" (UID: "322ef63c-182b-4ce2-aea7-8d666b625c19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:07:41.560240 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.560156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-dshm" (OuterVolumeSpecName: "dshm") pod "322ef63c-182b-4ce2-aea7-8d666b625c19" (UID: "322ef63c-182b-4ce2-aea7-8d666b625c19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:07:41.560240 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.560166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322ef63c-182b-4ce2-aea7-8d666b625c19-kube-api-access-8njsv" (OuterVolumeSpecName: "kube-api-access-8njsv") pod "322ef63c-182b-4ce2-aea7-8d666b625c19" (UID: "322ef63c-182b-4ce2-aea7-8d666b625c19"). InnerVolumeSpecName "kube-api-access-8njsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:07:41.659110 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.659074    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322ef63c-182b-4ce2-aea7-8d666b625c19-tls-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:41.659110 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.659102    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322ef63c-182b-4ce2-aea7-8d666b625c19-dshm\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:41.659110 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.659111    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8njsv\" (UniqueName: \"kubernetes.io/projected/322ef63c-182b-4ce2-aea7-8d666b625c19-kube-api-access-8njsv\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:41.927040 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.926998    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"]
Sep 09 13:07:41.929384 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:41.929360    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4b96-xg8f6"]
Sep 09 13:07:42.107789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:42.107752    2580 generic.go:356] "Generic (PLEG): container finished" podID="322ef63c-182b-4ce2-aea7-8d666b625c19" containerID="2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22" exitCode=0
Sep 09 13:07:42.108001 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:42.107797    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"
Sep 09 13:07:42.108001 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:42.107846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9" event={"ID":"322ef63c-182b-4ce2-aea7-8d666b625c19","Type":"ContainerDied","Data":"2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22"}
Sep 09 13:07:42.108001 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:42.107890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9" event={"ID":"322ef63c-182b-4ce2-aea7-8d666b625c19","Type":"ContainerDied","Data":"595d2e4a074346d8ecf06cd789373942e13ed5b4d76031fd0b65a414747faa9e"}
Sep 09 13:07:42.108001 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:42.107907    2580 scope.go:117] "RemoveContainer" containerID="2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22"
Sep 09 13:07:42.117049 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:42.117034    2580 scope.go:117] "RemoveContainer" containerID="2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22"
Sep 09 13:07:42.117311 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:07:42.117293    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22\": container with ID starting with 2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22 not found: ID does not exist" containerID="2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22"
Sep 09 13:07:42.117368 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:42.117319    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22"} err="failed to get container status \"2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22\": rpc error: code = NotFound desc = could not find container \"2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22\": container with ID starting with 2022f048e2efc55f38f1a9fe09e3eb61c4e42b9933d63d6fb1791de22ab92b22 not found: ID does not exist"
Sep 09 13:07:42.124174 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:42.124149    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"]
Sep 09 13:07:42.127813 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:42.127792    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-11f768/noten-model-11f768-kserve-6fd64b98dc-fbxd9"]
Sep 09 13:07:43.592368 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:43.592337    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322ef63c-182b-4ce2-aea7-8d666b625c19" path="/var/lib/kubelet/pods/322ef63c-182b-4ce2-aea7-8d666b625c19/volumes"
Sep 09 13:07:43.592741 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:43.592678    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa94e944-c387-4939-993a-49a62555d5be" path="/var/lib/kubelet/pods/fa94e944-c387-4939-993a-49a62555d5be/volumes"
Sep 09 13:07:47.172808 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:47.172774    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:07:58.414343 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.414299    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"]
Sep 09 13:07:58.415027 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.414606    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9" podUID="2e319adc-3828-4aea-8078-c5c28a632053" containerName="maas-api" containerID="cri-o://09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717" gracePeriod=30
Sep 09 13:07:58.671981 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.671923    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:58.831210 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.831174    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-metrics-certs\") pod \"2e319adc-3828-4aea-8078-c5c28a632053\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") "
Sep 09 13:07:58.831384 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.831255    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-maas-api-tls\") pod \"2e319adc-3828-4aea-8078-c5c28a632053\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") "
Sep 09 13:07:58.831384 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.831303    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e319adc-3828-4aea-8078-c5c28a632053-openshift-service-ca\") pod \"2e319adc-3828-4aea-8078-c5c28a632053\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") "
Sep 09 13:07:58.831384 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.831339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q8tb\" (UniqueName: \"kubernetes.io/projected/2e319adc-3828-4aea-8078-c5c28a632053-kube-api-access-9q8tb\") pod \"2e319adc-3828-4aea-8078-c5c28a632053\" (UID: \"2e319adc-3828-4aea-8078-c5c28a632053\") "
Sep 09 13:07:58.831719 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.831695    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e319adc-3828-4aea-8078-c5c28a632053-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e319adc-3828-4aea-8078-c5c28a632053" (UID: "2e319adc-3828-4aea-8078-c5c28a632053"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:07:58.833645 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.833616    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e319adc-3828-4aea-8078-c5c28a632053" (UID: "2e319adc-3828-4aea-8078-c5c28a632053"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:07:58.833767 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.833673    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e319adc-3828-4aea-8078-c5c28a632053" (UID: "2e319adc-3828-4aea-8078-c5c28a632053"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:07:58.833767 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.833700    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e319adc-3828-4aea-8078-c5c28a632053-kube-api-access-9q8tb" (OuterVolumeSpecName: "kube-api-access-9q8tb") pod "2e319adc-3828-4aea-8078-c5c28a632053" (UID: "2e319adc-3828-4aea-8078-c5c28a632053"). InnerVolumeSpecName "kube-api-access-9q8tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:07:58.932336 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.932227    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-metrics-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:58.932336 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.932278    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e319adc-3828-4aea-8078-c5c28a632053-maas-api-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:58.932336 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.932287    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e319adc-3828-4aea-8078-c5c28a632053-openshift-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:58.932336 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:58.932297    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q8tb\" (UniqueName: \"kubernetes.io/projected/2e319adc-3828-4aea-8078-c5c28a632053-kube-api-access-9q8tb\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:07:59.202020 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:59.201914    2580 generic.go:356] "Generic (PLEG): container finished" podID="2e319adc-3828-4aea-8078-c5c28a632053" containerID="09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717" exitCode=0
Sep 09 13:07:59.202020 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:59.201958    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"
Sep 09 13:07:59.202020 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:59.202006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9" event={"ID":"2e319adc-3828-4aea-8078-c5c28a632053","Type":"ContainerDied","Data":"09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717"}
Sep 09 13:07:59.202285 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:59.202057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9" event={"ID":"2e319adc-3828-4aea-8078-c5c28a632053","Type":"ContainerDied","Data":"bb4f8186d2e553a86191a7ffaade9117627e021890f84ad08b4fb1f4f7e8b77f"}
Sep 09 13:07:59.202285 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:59.202080    2580 scope.go:117] "RemoveContainer" containerID="09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717"
Sep 09 13:07:59.211675 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:59.211659    2580 scope.go:117] "RemoveContainer" containerID="09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717"
Sep 09 13:07:59.211986 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:07:59.211965    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717\": container with ID starting with 09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717 not found: ID does not exist" containerID="09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717"
Sep 09 13:07:59.212037 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:59.211995    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717"} err="failed to get container status \"09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717\": rpc error: code = NotFound desc = could not find container \"09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717\": container with ID starting with 09b51cf855855d0da691445fa0409f2a8b63e1ea0599cfe08e6f20d8fd3f2717 not found: ID does not exist"
Sep 09 13:07:59.231793 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:59.231771    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"]
Sep 09 13:07:59.244220 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:59.244196    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73011135-ff74bfb59-wg5m9"]
Sep 09 13:07:59.592242 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:07:59.592207    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e319adc-3828-4aea-8078-c5c28a632053" path="/var/lib/kubelet/pods/2e319adc-3828-4aea-8078-c5c28a632053/volumes"
Sep 09 13:08:00.198540 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:00.198492    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"]
Sep 09 13:08:00.198783 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:00.198757    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth" podUID="a80bba08-fd5a-4070-9511-aab8692e0681" containerName="payload-processing" containerID="cri-o://db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e" gracePeriod=30
Sep 09 13:08:00.377690 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:00.377662    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:08:00.466759 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:00.466732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:08:00.650007 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:00.649959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80bba08-fd5a-4070-9511-aab8692e0681-plugins-config-volume\") pod \"a80bba08-fd5a-4070-9511-aab8692e0681\" (UID: \"a80bba08-fd5a-4070-9511-aab8692e0681\") "
Sep 09 13:08:00.650415 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:00.650113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48h4d\" (UniqueName: \"kubernetes.io/projected/a80bba08-fd5a-4070-9511-aab8692e0681-kube-api-access-48h4d\") pod \"a80bba08-fd5a-4070-9511-aab8692e0681\" (UID: \"a80bba08-fd5a-4070-9511-aab8692e0681\") "
Sep 09 13:08:00.650415 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:00.650329    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80bba08-fd5a-4070-9511-aab8692e0681-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a80bba08-fd5a-4070-9511-aab8692e0681" (UID: "a80bba08-fd5a-4070-9511-aab8692e0681"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:08:00.652400 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:00.652380    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80bba08-fd5a-4070-9511-aab8692e0681-kube-api-access-48h4d" (OuterVolumeSpecName: "kube-api-access-48h4d") pod "a80bba08-fd5a-4070-9511-aab8692e0681" (UID: "a80bba08-fd5a-4070-9511-aab8692e0681"). InnerVolumeSpecName "kube-api-access-48h4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:08:00.750976 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:00.750892    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48h4d\" (UniqueName: \"kubernetes.io/projected/a80bba08-fd5a-4070-9511-aab8692e0681-kube-api-access-48h4d\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:08:00.750976 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:00.750918    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80bba08-fd5a-4070-9511-aab8692e0681-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:08:01.217834 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:01.217795    2580 generic.go:356] "Generic (PLEG): container finished" podID="a80bba08-fd5a-4070-9511-aab8692e0681" containerID="db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e" exitCode=0
Sep 09 13:08:01.218059 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:01.217864    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"
Sep 09 13:08:01.218059 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:01.217885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth" event={"ID":"a80bba08-fd5a-4070-9511-aab8692e0681","Type":"ContainerDied","Data":"db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e"}
Sep 09 13:08:01.218059 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:01.217953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth" event={"ID":"a80bba08-fd5a-4070-9511-aab8692e0681","Type":"ContainerDied","Data":"912b3e12e3a0e6c44945c854410fb12a97720864d9222ab0c013739f78969971"}
Sep 09 13:08:01.218059 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:01.217984    2580 scope.go:117] "RemoveContainer" containerID="db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e"
Sep 09 13:08:01.227483 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:01.227467    2580 scope.go:117] "RemoveContainer" containerID="db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e"
Sep 09 13:08:01.227751 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:08:01.227732    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e\": container with ID starting with db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e not found: ID does not exist" containerID="db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e"
Sep 09 13:08:01.227798 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:01.227761    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e"} err="failed to get container status \"db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e\": rpc error: code = NotFound desc = could not find container \"db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e\": container with ID starting with db7b109e24f89e4854561d69eaf2efdfbbd77546ceccc2779bbfb899bf5a505e not found: ID does not exist"
Sep 09 13:08:01.238957 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:01.238934    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"]
Sep 09 13:08:01.244913 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:01.244884    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c991b-678bfb965-n8kth"]
Sep 09 13:08:01.591968 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:01.591926    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a80bba08-fd5a-4070-9511-aab8692e0681" path="/var/lib/kubelet/pods/a80bba08-fd5a-4070-9511-aab8692e0681/volumes"
Sep 09 13:08:07.386451 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:07.386413    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"]
Sep 09 13:08:07.387003 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:07.386674    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy" containerID="cri-o://441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581" gracePeriod=30
Sep 09 13:08:07.576890 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:07.576859    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:08:09.046371 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:09.046334    2580 patch_prober.go:28] interesting pod/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 13:08:09.046736 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:09.046405    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 13:08:12.648756 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.648729    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:08:12.761083 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761002    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-istio-token\") pod \"02b2dea0-ebbb-4278-accb-c21559060609\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") "
Sep 09 13:08:12.761083 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lbts\" (UniqueName: \"kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-kube-api-access-9lbts\") pod \"02b2dea0-ebbb-4278-accb-c21559060609\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") "
Sep 09 13:08:12.761326 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761087    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b2dea0-ebbb-4278-accb-c21559060609-istio-podinfo\") pod \"02b2dea0-ebbb-4278-accb-c21559060609\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") "
Sep 09 13:08:12.761326 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-certs\") pod \"02b2dea0-ebbb-4278-accb-c21559060609\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") "
Sep 09 13:08:12.761326 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761131    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-envoy\") pod \"02b2dea0-ebbb-4278-accb-c21559060609\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") "
Sep 09 13:08:12.761326 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-credential-socket\") pod \"02b2dea0-ebbb-4278-accb-c21559060609\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") "
Sep 09 13:08:12.761326 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b2dea0-ebbb-4278-accb-c21559060609-istiod-ca-cert\") pod \"02b2dea0-ebbb-4278-accb-c21559060609\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") "
Sep 09 13:08:12.761326 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-socket\") pod \"02b2dea0-ebbb-4278-accb-c21559060609\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") "
Sep 09 13:08:12.761326 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761310    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-data\") pod \"02b2dea0-ebbb-4278-accb-c21559060609\" (UID: \"02b2dea0-ebbb-4278-accb-c21559060609\") "
Sep 09 13:08:12.761704 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761562    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02b2dea0-ebbb-4278-accb-c21559060609" (UID: "02b2dea0-ebbb-4278-accb-c21559060609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:08:12.761704 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761646    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02b2dea0-ebbb-4278-accb-c21559060609" (UID: "02b2dea0-ebbb-4278-accb-c21559060609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:08:12.761704 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761666    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-credential-socket\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:08:12.761945 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761846    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02b2dea0-ebbb-4278-accb-c21559060609" (UID: "02b2dea0-ebbb-4278-accb-c21559060609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:08:12.762025 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.761918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b2dea0-ebbb-4278-accb-c21559060609-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02b2dea0-ebbb-4278-accb-c21559060609" (UID: "02b2dea0-ebbb-4278-accb-c21559060609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:08:12.762175 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.762138    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-data" (OuterVolumeSpecName: "istio-data") pod "02b2dea0-ebbb-4278-accb-c21559060609" (UID: "02b2dea0-ebbb-4278-accb-c21559060609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:08:12.763499 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.763480    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02b2dea0-ebbb-4278-accb-c21559060609-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02b2dea0-ebbb-4278-accb-c21559060609" (UID: "02b2dea0-ebbb-4278-accb-c21559060609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:08:12.763637 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.763619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-kube-api-access-9lbts" (OuterVolumeSpecName: "kube-api-access-9lbts") pod "02b2dea0-ebbb-4278-accb-c21559060609" (UID: "02b2dea0-ebbb-4278-accb-c21559060609"). InnerVolumeSpecName "kube-api-access-9lbts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:08:12.763985 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.763965    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-istio-token" (OuterVolumeSpecName: "istio-token") pod "02b2dea0-ebbb-4278-accb-c21559060609" (UID: "02b2dea0-ebbb-4278-accb-c21559060609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:08:12.763985 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.763971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02b2dea0-ebbb-4278-accb-c21559060609" (UID: "02b2dea0-ebbb-4278-accb-c21559060609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:08:12.862866 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.862804    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b2dea0-ebbb-4278-accb-c21559060609-istiod-ca-cert\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:08:12.862866 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.862857    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-socket\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:08:12.862866 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.862867    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-data\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:08:12.862866 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.862876    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-istio-token\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:08:12.863132 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.862884    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lbts\" (UniqueName: \"kubernetes.io/projected/02b2dea0-ebbb-4278-accb-c21559060609-kube-api-access-9lbts\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:08:12.863132 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.862894    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b2dea0-ebbb-4278-accb-c21559060609-istio-podinfo\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:08:12.863132 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.862902    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-workload-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:08:12.863132 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:12.862910    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b2dea0-ebbb-4278-accb-c21559060609-istio-envoy\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:08:13.283588 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.283553    2580 generic.go:356] "Generic (PLEG): container finished" podID="02b2dea0-ebbb-4278-accb-c21559060609" containerID="441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581" exitCode=0
Sep 09 13:08:13.283866 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.283607    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"
Sep 09 13:08:13.283866 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.283625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" event={"ID":"02b2dea0-ebbb-4278-accb-c21559060609","Type":"ContainerDied","Data":"441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581"}
Sep 09 13:08:13.283866 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.283681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd" event={"ID":"02b2dea0-ebbb-4278-accb-c21559060609","Type":"ContainerDied","Data":"b8c5cb4ca7d73110378a96cfe929e04e448821b2c44c74ff766ab5d8d1b74cb3"}
Sep 09 13:08:13.283866 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.283709    2580 scope.go:117] "RemoveContainer" containerID="441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581"
Sep 09 13:08:13.294011 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.293980    2580 scope.go:117] "RemoveContainer" containerID="85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2"
Sep 09 13:08:13.303652 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.303627    2580 scope.go:117] "RemoveContainer" containerID="441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581"
Sep 09 13:08:13.304063 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:08:13.304037    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581\": container with ID starting with 441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581 not found: ID does not exist" containerID="441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581"
Sep 09 13:08:13.304163 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.304072    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581"} err="failed to get container status \"441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581\": rpc error: code = NotFound desc = could not find container \"441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581\": container with ID starting with 441d3f3fc93a919990ed62168b662864bcc24a1d2adfe3c008760be8636ce581 not found: ID does not exist"
Sep 09 13:08:13.304163 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.304099    2580 scope.go:117] "RemoveContainer" containerID="85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2"
Sep 09 13:08:13.304387 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:08:13.304368    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2\": container with ID starting with 85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2 not found: ID does not exist" containerID="85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2"
Sep 09 13:08:13.304438 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.304396    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2"} err="failed to get container status \"85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2\": rpc error: code = NotFound desc = could not find container \"85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2\": container with ID starting with 85c5bc92a5ae77d5e5c13ca4ab521faf6306a1e8351c88a4ec2c565ec99394f2 not found: ID does not exist"
Sep 09 13:08:13.306193 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.306171    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"]
Sep 09 13:08:13.309429 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.309409    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7c991b-openshift-default-7868bbccbb-llzkd"]
Sep 09 13:08:13.592418 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.592386    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02b2dea0-ebbb-4278-accb-c21559060609" path="/var/lib/kubelet/pods/02b2dea0-ebbb-4278-accb-c21559060609/volumes"
Sep 09 13:08:13.653111 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653080    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"]
Sep 09 13:08:13.653508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653466    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy"
Sep 09 13:08:13.653508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653478    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy"
Sep 09 13:08:13.653508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653493    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy"
Sep 09 13:08:13.653508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653499    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653512    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322ef63c-182b-4ce2-aea7-8d666b625c19" containerName="main"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653518    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="322ef63c-182b-4ce2-aea7-8d666b625c19" containerName="main"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653524    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa94e944-c387-4939-993a-49a62555d5be" containerName="revoke-keys"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653529    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa94e944-c387-4939-993a-49a62555d5be" containerName="revoke-keys"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653543    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11617c63-0e15-4583-97f4-e459e7b8a413" containerName="istio-proxy"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653547    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11617c63-0e15-4583-97f4-e459e7b8a413" containerName="istio-proxy"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653558    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a80bba08-fd5a-4070-9511-aab8692e0681" containerName="payload-processing"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653563    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a80bba08-fd5a-4070-9511-aab8692e0681" containerName="payload-processing"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653573    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e319adc-3828-4aea-8078-c5c28a632053" containerName="maas-api"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653578    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e319adc-3828-4aea-8078-c5c28a632053" containerName="maas-api"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653632    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e319adc-3828-4aea-8078-c5c28a632053" containerName="maas-api"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653640    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy"
Sep 09 13:08:13.653642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653646    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="11617c63-0e15-4583-97f4-e459e7b8a413" containerName="istio-proxy"
Sep 09 13:08:13.654046 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653652    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="322ef63c-182b-4ce2-aea7-8d666b625c19" containerName="main"
Sep 09 13:08:13.654046 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653659    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a80bba08-fd5a-4070-9511-aab8692e0681" containerName="payload-processing"
Sep 09 13:08:13.654046 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.653666    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa94e944-c387-4939-993a-49a62555d5be" containerName="revoke-keys"
Sep 09 13:08:13.658369 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.658352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:08:13.660869 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.660843    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ada1ae-dockercfg-7jzdk\""
Sep 09 13:08:13.661012 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.660892    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ada1ae\""
Sep 09 13:08:13.667228 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.667204    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"]
Sep 09 13:08:13.773348 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.773298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5jx7\" (UniqueName: \"kubernetes.io/projected/be0a2021-5883-44d5-b45f-a2d5af66f873-kube-api-access-g5jx7\") pod \"payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9\" (UID: \"be0a2021-5883-44d5-b45f-a2d5af66f873\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:08:13.773540 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.773433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0a2021-5883-44d5-b45f-a2d5af66f873-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9\" (UID: \"be0a2021-5883-44d5-b45f-a2d5af66f873\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:08:13.874356 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.874250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0a2021-5883-44d5-b45f-a2d5af66f873-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9\" (UID: \"be0a2021-5883-44d5-b45f-a2d5af66f873\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:08:13.874510 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.874376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5jx7\" (UniqueName: \"kubernetes.io/projected/be0a2021-5883-44d5-b45f-a2d5af66f873-kube-api-access-g5jx7\") pod \"payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9\" (UID: \"be0a2021-5883-44d5-b45f-a2d5af66f873\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:08:13.875052 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.875026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0a2021-5883-44d5-b45f-a2d5af66f873-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9\" (UID: \"be0a2021-5883-44d5-b45f-a2d5af66f873\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:08:13.883547 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.883523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5jx7\" (UniqueName: \"kubernetes.io/projected/be0a2021-5883-44d5-b45f-a2d5af66f873-kube-api-access-g5jx7\") pod \"payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9\" (UID: \"be0a2021-5883-44d5-b45f-a2d5af66f873\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:08:13.969969 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:13.969931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:08:14.245192 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:14.245099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"]
Sep 09 13:08:14.250957 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:08:14.250932    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0a2021_5883_44d5_b45f_a2d5af66f873.slice/crio-3e1406786306520b0982e29437f768afaec6d90cd9ec3f6dba9120ba06f588bb WatchSource:0}: Error finding container 3e1406786306520b0982e29437f768afaec6d90cd9ec3f6dba9120ba06f588bb: Status 404 returned error can't find the container with id 3e1406786306520b0982e29437f768afaec6d90cd9ec3f6dba9120ba06f588bb
Sep 09 13:08:14.291433 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:14.291400    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9" event={"ID":"be0a2021-5883-44d5-b45f-a2d5af66f873","Type":"ContainerStarted","Data":"3e1406786306520b0982e29437f768afaec6d90cd9ec3f6dba9120ba06f588bb"}
Sep 09 13:08:15.297998 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:15.297958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9" event={"ID":"be0a2021-5883-44d5-b45f-a2d5af66f873","Type":"ContainerStarted","Data":"cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2"}
Sep 09 13:08:15.298426 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:15.298113    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:08:15.315412 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:15.315360    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9" podStartSLOduration=2.315345839 podStartE2EDuration="2.315345839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:08:15.31370047 +0000 UTC m=+1178.233591811" watchObservedRunningTime="2026-09-09 13:08:15.315345839 +0000 UTC m=+1178.235237178"
Sep 09 13:08:16.168160 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:16.168126    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:08:23.661903 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.661864    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"]
Sep 09 13:08:23.662327 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.662314    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b2dea0-ebbb-4278-accb-c21559060609" containerName="istio-proxy"
Sep 09 13:08:23.665541 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.665520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:08:23.668799 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.668774    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-0b0a1a-dockercfg-9j8kh\""
Sep 09 13:08:23.669068 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.669053    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-0b0a1a\""
Sep 09 13:08:23.676626 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.676595    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"]
Sep 09 13:08:23.700442 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.700403    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"]
Sep 09 13:08:23.704085 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.704061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:08:23.715305 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.715278    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"]
Sep 09 13:08:23.770529 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.770493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7456ae98-5004-4a87-817d-b39def859cb7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl\" (UID: \"7456ae98-5004-4a87-817d-b39def859cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:08:23.770697 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.770538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znvzm\" (UniqueName: \"kubernetes.io/projected/7456ae98-5004-4a87-817d-b39def859cb7-kube-api-access-znvzm\") pod \"payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl\" (UID: \"7456ae98-5004-4a87-817d-b39def859cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:08:23.770697 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.770600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4bs9\" (UniqueName: \"kubernetes.io/projected/23cbe1ad-69c6-4592-b104-5fec49466934-kube-api-access-h4bs9\") pod \"payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j\" (UID: \"23cbe1ad-69c6-4592-b104-5fec49466934\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:08:23.770790 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.770705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23cbe1ad-69c6-4592-b104-5fec49466934-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j\" (UID: \"23cbe1ad-69c6-4592-b104-5fec49466934\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:08:23.871602 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.871554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7456ae98-5004-4a87-817d-b39def859cb7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl\" (UID: \"7456ae98-5004-4a87-817d-b39def859cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:08:23.871858 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.871616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znvzm\" (UniqueName: \"kubernetes.io/projected/7456ae98-5004-4a87-817d-b39def859cb7-kube-api-access-znvzm\") pod \"payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl\" (UID: \"7456ae98-5004-4a87-817d-b39def859cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:08:23.871858 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.871752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4bs9\" (UniqueName: \"kubernetes.io/projected/23cbe1ad-69c6-4592-b104-5fec49466934-kube-api-access-h4bs9\") pod \"payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j\" (UID: \"23cbe1ad-69c6-4592-b104-5fec49466934\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:08:23.871989 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.871911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23cbe1ad-69c6-4592-b104-5fec49466934-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j\" (UID: \"23cbe1ad-69c6-4592-b104-5fec49466934\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:08:23.872519 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.872495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7456ae98-5004-4a87-817d-b39def859cb7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl\" (UID: \"7456ae98-5004-4a87-817d-b39def859cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:08:23.872586 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.872569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23cbe1ad-69c6-4592-b104-5fec49466934-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j\" (UID: \"23cbe1ad-69c6-4592-b104-5fec49466934\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:08:23.880847 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.880800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znvzm\" (UniqueName: \"kubernetes.io/projected/7456ae98-5004-4a87-817d-b39def859cb7-kube-api-access-znvzm\") pod \"payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl\" (UID: \"7456ae98-5004-4a87-817d-b39def859cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:08:23.880980 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.880798    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4bs9\" (UniqueName: \"kubernetes.io/projected/23cbe1ad-69c6-4592-b104-5fec49466934-kube-api-access-h4bs9\") pod \"payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j\" (UID: \"23cbe1ad-69c6-4592-b104-5fec49466934\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:08:23.977987 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:23.977893    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:08:24.014475 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:24.014435    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:08:24.262747 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:24.262693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"]
Sep 09 13:08:24.265076 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:08:24.265050    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cbe1ad_69c6_4592_b104_5fec49466934.slice/crio-a4a4c700d561010fc9846e51cd0d02e468c008ca199e03d8dd38d965f2594df9 WatchSource:0}: Error finding container a4a4c700d561010fc9846e51cd0d02e468c008ca199e03d8dd38d965f2594df9: Status 404 returned error can't find the container with id a4a4c700d561010fc9846e51cd0d02e468c008ca199e03d8dd38d965f2594df9
Sep 09 13:08:24.301675 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:24.301650    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"]
Sep 09 13:08:24.303545 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:08:24.303518    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7456ae98_5004_4a87_817d_b39def859cb7.slice/crio-ecbab288f185f45ee72d92408f9eb4f34d912e47a5ed9674f3c106adaa146e84 WatchSource:0}: Error finding container ecbab288f185f45ee72d92408f9eb4f34d912e47a5ed9674f3c106adaa146e84: Status 404 returned error can't find the container with id ecbab288f185f45ee72d92408f9eb4f34d912e47a5ed9674f3c106adaa146e84
Sep 09 13:08:24.348808 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:24.348747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl" event={"ID":"7456ae98-5004-4a87-817d-b39def859cb7","Type":"ContainerStarted","Data":"ecbab288f185f45ee72d92408f9eb4f34d912e47a5ed9674f3c106adaa146e84"}
Sep 09 13:08:24.350655 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:24.350629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j" event={"ID":"23cbe1ad-69c6-4592-b104-5fec49466934","Type":"ContainerStarted","Data":"c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97"}
Sep 09 13:08:24.350807 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:24.350675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j" event={"ID":"23cbe1ad-69c6-4592-b104-5fec49466934","Type":"ContainerStarted","Data":"a4a4c700d561010fc9846e51cd0d02e468c008ca199e03d8dd38d965f2594df9"}
Sep 09 13:08:24.350807 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:24.350711    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:08:24.368405 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:24.368354    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j" podStartSLOduration=1.368335128 podStartE2EDuration="1.368335128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:08:24.367061869 +0000 UTC m=+1187.286953208" watchObservedRunningTime="2026-09-09 13:08:24.368335128 +0000 UTC m=+1187.288226524"
Sep 09 13:08:25.357534 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:25.357498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl" event={"ID":"7456ae98-5004-4a87-817d-b39def859cb7","Type":"ContainerStarted","Data":"848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda"}
Sep 09 13:08:25.357992 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:25.357568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:08:25.374153 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:25.374094    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl" podStartSLOduration=2.37407948 podStartE2EDuration="2.37407948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:08:25.372472072 +0000 UTC m=+1188.292363411" watchObservedRunningTime="2026-09-09 13:08:25.37407948 +0000 UTC m=+1188.293970820"
Sep 09 13:08:26.305161 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:26.305128    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:08:27.078041 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:27.078004    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:08:35.360013 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:35.359983    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:08:36.365514 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:36.365482    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:08:38.153707 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:38.153678    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 13:08:38.156970 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:38.156949    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 13:08:42.099753 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.099718    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"]
Sep 09 13:08:42.105024 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.104984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.108460 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.108422    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 13:08:42.108460 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.108448    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:08:42.109098 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.109072    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 13:08:42.109340 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.109077    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-650a99-serving-cert\""
Sep 09 13:08:42.109446 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.109111    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dbjk4\""
Sep 09 13:08:42.114132 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.114092    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"]
Sep 09 13:08:42.152386 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.152345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.152575 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.152404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78ww\" (UniqueName: \"kubernetes.io/projected/388cde2e-18cb-44cb-b4c4-7e3726160f01-kube-api-access-n78ww\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.152575 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.152478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388cde2e-18cb-44cb-b4c4-7e3726160f01-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.152575 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.152527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.254016 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.253967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78ww\" (UniqueName: \"kubernetes.io/projected/388cde2e-18cb-44cb-b4c4-7e3726160f01-kube-api-access-n78ww\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.254203 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.254060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388cde2e-18cb-44cb-b4c4-7e3726160f01-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.254203 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.254133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.254203 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.254195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.254340 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:08:42.254277    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-serving-cert: secret "maas-api-e2e-shared-b-w3-650a99-serving-cert" not found
Sep 09 13:08:42.254383 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:08:42.254366    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-maas-api-tls podName:388cde2e-18cb-44cb-b4c4-7e3726160f01 nodeName:}" failed. No retries permitted until 2026-09-09 13:08:42.754342612 +0000 UTC m=+1205.674233938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-maas-api-tls") pod "maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x" (UID: "388cde2e-18cb-44cb-b4c4-7e3726160f01") : secret "maas-api-e2e-shared-b-w3-650a99-serving-cert" not found
Sep 09 13:08:42.254790 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.254773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388cde2e-18cb-44cb-b4c4-7e3726160f01-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.256806 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.256790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.262377 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.262353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78ww\" (UniqueName: \"kubernetes.io/projected/388cde2e-18cb-44cb-b4c4-7e3726160f01-kube-api-access-n78ww\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.739801 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.739764    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"]
Sep 09 13:08:42.743579 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.743558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:08:42.746261 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.746241    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-650a99\""
Sep 09 13:08:42.746531 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.746515    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-650a99-dockercfg-l7fcs\""
Sep 09 13:08:42.755109 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.755081    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"]
Sep 09 13:08:42.759111 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.759083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4a1b42-c978-4957-96b9-efa47269c720-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x\" (UID: \"af4a1b42-c978-4957-96b9-efa47269c720\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:08:42.759211 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.759143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.759289 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.759273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfl4d\" (UniqueName: \"kubernetes.io/projected/af4a1b42-c978-4957-96b9-efa47269c720-kube-api-access-pfl4d\") pod \"payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x\" (UID: \"af4a1b42-c978-4957-96b9-efa47269c720\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:08:42.761616 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.761588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:42.860135 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.860092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4a1b42-c978-4957-96b9-efa47269c720-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x\" (UID: \"af4a1b42-c978-4957-96b9-efa47269c720\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:08:42.860352 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.860213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfl4d\" (UniqueName: \"kubernetes.io/projected/af4a1b42-c978-4957-96b9-efa47269c720-kube-api-access-pfl4d\") pod \"payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x\" (UID: \"af4a1b42-c978-4957-96b9-efa47269c720\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:08:42.860882 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.860859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4a1b42-c978-4957-96b9-efa47269c720-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x\" (UID: \"af4a1b42-c978-4957-96b9-efa47269c720\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:08:42.868152 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:42.868129    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfl4d\" (UniqueName: \"kubernetes.io/projected/af4a1b42-c978-4957-96b9-efa47269c720-kube-api-access-pfl4d\") pod \"payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x\" (UID: \"af4a1b42-c978-4957-96b9-efa47269c720\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:08:43.024187 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:43.024094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:43.055013 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:43.054976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:08:43.299735 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:43.299709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"]
Sep 09 13:08:43.301684 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:08:43.301658    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388cde2e_18cb_44cb_b4c4_7e3726160f01.slice/crio-3c54afc12a40e091474f34246bf4c26cccfc37ab6be7de4e718ffa527d6bf02d WatchSource:0}: Error finding container 3c54afc12a40e091474f34246bf4c26cccfc37ab6be7de4e718ffa527d6bf02d: Status 404 returned error can't find the container with id 3c54afc12a40e091474f34246bf4c26cccfc37ab6be7de4e718ffa527d6bf02d
Sep 09 13:08:43.339793 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:43.339767    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"]
Sep 09 13:08:43.342021 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:08:43.341989    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4a1b42_c978_4957_96b9_efa47269c720.slice/crio-0560b3d83e9b036494b14b88fd5017b9d33e3f2d26bc694df7d4383fa3536ab7 WatchSource:0}: Error finding container 0560b3d83e9b036494b14b88fd5017b9d33e3f2d26bc694df7d4383fa3536ab7: Status 404 returned error can't find the container with id 0560b3d83e9b036494b14b88fd5017b9d33e3f2d26bc694df7d4383fa3536ab7
Sep 09 13:08:43.467436 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:43.467392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x" event={"ID":"388cde2e-18cb-44cb-b4c4-7e3726160f01","Type":"ContainerStarted","Data":"3c54afc12a40e091474f34246bf4c26cccfc37ab6be7de4e718ffa527d6bf02d"}
Sep 09 13:08:43.469089 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:43.469047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x" event={"ID":"af4a1b42-c978-4957-96b9-efa47269c720","Type":"ContainerStarted","Data":"f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5"}
Sep 09 13:08:43.469235 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:43.469099    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x" event={"ID":"af4a1b42-c978-4957-96b9-efa47269c720","Type":"ContainerStarted","Data":"0560b3d83e9b036494b14b88fd5017b9d33e3f2d26bc694df7d4383fa3536ab7"}
Sep 09 13:08:43.469235 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:43.469127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:08:43.486678 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:43.486618    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x" podStartSLOduration=1.486595271 podStartE2EDuration="1.486595271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:08:43.48419434 +0000 UTC m=+1206.404085680" watchObservedRunningTime="2026-09-09 13:08:43.486595271 +0000 UTC m=+1206.406486610"
Sep 09 13:08:44.475864 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:44.475832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x" event={"ID":"388cde2e-18cb-44cb-b4c4-7e3726160f01","Type":"ContainerStarted","Data":"2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986"}
Sep 09 13:08:44.476345 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:44.475977    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:44.494469 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:44.494422    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x" podStartSLOduration=2.131037342 podStartE2EDuration="2.494406368s" totalImagesPullingTime="363.369026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:08:44.492557326 +0000 UTC m=+1207.412448677" watchObservedRunningTime="2026-09-09 13:08:44.494406368 +0000 UTC m=+1207.414297706"
Sep 09 13:08:47.275238 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.275196    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:08:47.595663 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.595621    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"]
Sep 09 13:08:47.600130 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.600083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.602953 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.602914    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 13:08:47.603152 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.602966    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-941953-openshift-default-dockercfg-dsjcq\""
Sep 09 13:08:47.607591 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.607547    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"]
Sep 09 13:08:47.706789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.706753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-envoy\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.706789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.706791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqmk\" (UniqueName: \"kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-kube-api-access-rwqmk\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.707008 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.706814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-socket\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.707008 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.706918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-token\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.707008 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.706970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-podinfo\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.707106 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.707028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-certs\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.707106 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.707074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-credential-socket\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.707106 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.707097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istiod-ca-cert\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.707219 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.707147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-data\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808148 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-data\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808359 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-envoy\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808359 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqmk\" (UniqueName: \"kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-kube-api-access-rwqmk\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808359 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-socket\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808359 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-token\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808359 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-podinfo\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808632 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-certs\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808632 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-credential-socket\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808632 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istiod-ca-cert\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808632 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-data\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808853 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-socket\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.808917 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-certs\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.809135 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.808927    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-credential-socket\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.809190 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.809147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istiod-ca-cert\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.810676 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.810652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-envoy\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.811252 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.811232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-podinfo\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.816260 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.816230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqmk\" (UniqueName: \"kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-kube-api-access-rwqmk\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.816260 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.816240    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-token\") pod \"e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") " pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:47.917814 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:47.917714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:48.195828 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:48.195790    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"]
Sep 09 13:08:48.197369 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:08:48.197338    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be7e4c1_d44c_434f_9a3d_7e679ffd777b.slice/crio-138ff486cae3d9bdfd849f8737b6b27d66d4f3dd3a92c4eb81bfde5f82882509 WatchSource:0}: Error finding container 138ff486cae3d9bdfd849f8737b6b27d66d4f3dd3a92c4eb81bfde5f82882509: Status 404 returned error can't find the container with id 138ff486cae3d9bdfd849f8737b6b27d66d4f3dd3a92c4eb81bfde5f82882509
Sep 09 13:08:48.199761 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:48.199731    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:08:48.199853 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:48.199794    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:08:48.199853 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:48.199840    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:08:48.506806 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:48.506702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" event={"ID":"5be7e4c1-d44c-434f-9a3d-7e679ffd777b","Type":"ContainerStarted","Data":"ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1"}
Sep 09 13:08:48.506806 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:48.506761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" event={"ID":"5be7e4c1-d44c-434f-9a3d-7e679ffd777b","Type":"ContainerStarted","Data":"138ff486cae3d9bdfd849f8737b6b27d66d4f3dd3a92c4eb81bfde5f82882509"}
Sep 09 13:08:48.525598 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:48.525544    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" podStartSLOduration=1.525524442 podStartE2EDuration="1.525524442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:08:48.524063243 +0000 UTC m=+1211.443954583" watchObservedRunningTime="2026-09-09 13:08:48.525524442 +0000 UTC m=+1211.445415782"
Sep 09 13:08:48.918406 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:48.918372    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:49.923916 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:49.923882    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:50.486937 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:50.486904    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:08:50.517992 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:50.517962    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:50.519113 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:50.519088    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:08:54.477505 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:54.477472    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:08:57.269572 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:08:57.269533    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:09:06.561991 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:06.561949    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 13:09:06.562371 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:06.562023    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 13:09:07.613983 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:07.613949    2580 generic.go:356] "Generic (PLEG): container finished" podID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerID="ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1" exitCode=0
Sep 09 13:09:07.614421 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:07.614021    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" event={"ID":"5be7e4c1-d44c-434f-9a3d-7e679ffd777b","Type":"ContainerDied","Data":"ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1"}
Sep 09 13:09:07.614421 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:07.614391    2580 scope.go:117] "RemoveContainer" containerID="ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1"
Sep 09 13:09:07.615464 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:07.615439    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:09:07.615532 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:07.615506    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:09:07.615568 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:07.615535    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 13:09:08.625645 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:08.625610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" event={"ID":"5be7e4c1-d44c-434f-9a3d-7e679ffd777b","Type":"ContainerStarted","Data":"1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2"}
Sep 09 13:09:08.918838 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:08.918721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:09:08.919968 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:08.919123    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 13:09:08.920183 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:08.920158    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 13:09:09.919147 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:09.919101    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 13:09:09.919656 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:09.919173    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 13:09:10.919263 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:10.919214    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 13:09:10.919800 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:10.919288    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 13:09:11.919197 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:11.919158    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 13:09:11.919483 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:11.919454    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 13:09:12.918734 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:12.918693    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 13:09:12.919083 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:12.919040    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 13:09:13.371091 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:13.371051    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:09:13.922364 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:13.922328    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:09:13.922742 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:13.922720    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:09:13.923514 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:13.923495    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:09:30.457584 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.457497    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"]
Sep 09 13:09:30.461601 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.461581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.465256 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.465230    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-650a99\"/\"kube-root-ca.crt\""
Sep 09 13:09:30.465409 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.465237    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-650a99\"/\"openshift-service-ca.crt\""
Sep 09 13:09:30.465409 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.465237    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-650a99\"/\"auth-test-model-650a99-kserve-self-signed-certs\""
Sep 09 13:09:30.465409 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.465237    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-650a99\"/\"default-dockercfg-n72nf\""
Sep 09 13:09:30.470903 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.470879    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"]
Sep 09 13:09:30.536550 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.536514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-model-cache\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.536729 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.536581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-home\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.536729 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.536634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42683abf-51bd-4908-8906-b7a055b83e45-tls-certs\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.536729 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.536667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-dshm\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.536729 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.536689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tw59\" (UniqueName: \"kubernetes.io/projected/42683abf-51bd-4908-8906-b7a055b83e45-kube-api-access-8tw59\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.536928 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.536783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-tmp-dir\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.638340 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.638304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-tmp-dir\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.638544 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.638350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-model-cache\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.638544 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.638498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-home\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.638671 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.638592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42683abf-51bd-4908-8906-b7a055b83e45-tls-certs\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.638671 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.638649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-dshm\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.638773 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.638697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tw59\" (UniqueName: \"kubernetes.io/projected/42683abf-51bd-4908-8906-b7a055b83e45-kube-api-access-8tw59\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.638773 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.638730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-model-cache\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.638903 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.638876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-tmp-dir\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.638960 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.638918    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-home\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.641149 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.641128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-dshm\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.641319 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.641301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42683abf-51bd-4908-8906-b7a055b83e45-tls-certs\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.646456 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.646437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tw59\" (UniqueName: \"kubernetes.io/projected/42683abf-51bd-4908-8906-b7a055b83e45-kube-api-access-8tw59\") pod \"auth-test-model-650a99-kserve-f6568db6c-4mxkm\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") " pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:30.772985 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:30.772887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:31.044703 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:31.044678    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"]
Sep 09 13:09:31.046583 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:09:31.046554    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42683abf_51bd_4908_8906_b7a055b83e45.slice/crio-665d96a51ce690caba9cc0663c0163dd8f9ad3d176e3570001086c552b2c5a1b WatchSource:0}: Error finding container 665d96a51ce690caba9cc0663c0163dd8f9ad3d176e3570001086c552b2c5a1b: Status 404 returned error can't find the container with id 665d96a51ce690caba9cc0663c0163dd8f9ad3d176e3570001086c552b2c5a1b
Sep 09 13:09:31.761279 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:31.761232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm" event={"ID":"42683abf-51bd-4908-8906-b7a055b83e45","Type":"ContainerStarted","Data":"79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60"}
Sep 09 13:09:31.761279 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:31.761283    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm" event={"ID":"42683abf-51bd-4908-8906-b7a055b83e45","Type":"ContainerStarted","Data":"665d96a51ce690caba9cc0663c0163dd8f9ad3d176e3570001086c552b2c5a1b"}
Sep 09 13:09:31.779624 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:31.779573    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm" podStartSLOduration=1.779557796 podStartE2EDuration="1.779557796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:09:31.778912557 +0000 UTC m=+1254.698803896" watchObservedRunningTime="2026-09-09 13:09:31.779557796 +0000 UTC m=+1254.699449134"
Sep 09 13:09:35.971121 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:35.971087    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:09:36.483134 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.483099    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"]
Sep 09 13:09:36.485936 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.485914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:36.488339 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.488317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-597fffff-serving-cert\""
Sep 09 13:09:36.499491 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.499459    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"]
Sep 09 13:09:36.599832 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.599788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svg2k\" (UniqueName: \"kubernetes.io/projected/71b22cbf-2367-4b9e-9144-d6a7d89517a7-kube-api-access-svg2k\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:36.600011 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.599855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-maas-api-tls\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:36.600011 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.599876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-metrics-certs\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:36.600011 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.599976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b22cbf-2367-4b9e-9144-d6a7d89517a7-openshift-service-ca\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:36.700984 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.700942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b22cbf-2367-4b9e-9144-d6a7d89517a7-openshift-service-ca\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:36.701154 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.701046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svg2k\" (UniqueName: \"kubernetes.io/projected/71b22cbf-2367-4b9e-9144-d6a7d89517a7-kube-api-access-svg2k\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:36.701154 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.701094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-maas-api-tls\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:36.701154 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.701116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-metrics-certs\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:36.701360 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:09:36.701194    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-serving-cert: secret "maas-api-e2e-mt-597fffff-serving-cert" not found
Sep 09 13:09:36.701360 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:09:36.701259    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-maas-api-tls podName:71b22cbf-2367-4b9e-9144-d6a7d89517a7 nodeName:}" failed. No retries permitted until 2026-09-09 13:09:37.201236826 +0000 UTC m=+1260.121128163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-maas-api-tls") pod "maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2" (UID: "71b22cbf-2367-4b9e-9144-d6a7d89517a7") : secret "maas-api-e2e-mt-597fffff-serving-cert" not found
Sep 09 13:09:36.702117 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.702087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b22cbf-2367-4b9e-9144-d6a7d89517a7-openshift-service-ca\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:36.703791 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.703767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-metrics-certs\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:36.709357 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:36.709320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svg2k\" (UniqueName: \"kubernetes.io/projected/71b22cbf-2367-4b9e-9144-d6a7d89517a7-kube-api-access-svg2k\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:37.206840 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:37.206782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-maas-api-tls\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:37.209567 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:37.209529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-maas-api-tls\") pod \"maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:37.398302 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:37.398260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:37.674415 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:37.674385    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"]
Sep 09 13:09:37.675831 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:09:37.675780    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b22cbf_2367_4b9e_9144_d6a7d89517a7.slice/crio-f2766cc6c5f81754d4088a5075b7af87b2caa6af163f74d8a55d04631d361ac3 WatchSource:0}: Error finding container f2766cc6c5f81754d4088a5075b7af87b2caa6af163f74d8a55d04631d361ac3: Status 404 returned error can't find the container with id f2766cc6c5f81754d4088a5075b7af87b2caa6af163f74d8a55d04631d361ac3
Sep 09 13:09:37.798454 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:37.798413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2" event={"ID":"71b22cbf-2367-4b9e-9144-d6a7d89517a7","Type":"ContainerStarted","Data":"f2766cc6c5f81754d4088a5075b7af87b2caa6af163f74d8a55d04631d361ac3"}
Sep 09 13:09:38.805626 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:38.805585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2" event={"ID":"71b22cbf-2367-4b9e-9144-d6a7d89517a7","Type":"ContainerStarted","Data":"a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6"}
Sep 09 13:09:38.806031 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:38.805662    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:38.825342 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:38.825284    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2" podStartSLOduration=2.399591298 podStartE2EDuration="2.825265807s" totalImagesPullingTime="425.674509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:09:38.822654522 +0000 UTC m=+1261.742545860" watchObservedRunningTime="2026-09-09 13:09:38.825265807 +0000 UTC m=+1261.745157179"
Sep 09 13:09:40.774016 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:40.773973    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:40.774432 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:40.774229    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:40.786906 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:40.786876    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:40.827190 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:40.827157    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:09:44.816612 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:44.816583    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:09:53.164119 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:09:53.164086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:10:07.467668 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:10:07.467631    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:10:25.466910 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:10:25.466869    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:10:37.689665 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:10:37.689631    2580 scope.go:117] "RemoveContainer" containerID="c3c6f881742a7c2769e6111110880a999f88638cc4bd48fb0d1b69edf67b824f"
Sep 09 13:10:37.698852 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:10:37.698813    2580 scope.go:117] "RemoveContainer" containerID="e1ca47866070096f444dc72b5827183f85cfa513bfa014e6df0415516967b34c"
Sep 09 13:10:48.069211 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:10:48.069179    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:11:00.586345 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:00.586246    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"]
Sep 09 13:11:00.586840 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:00.586500    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2" podUID="71b22cbf-2367-4b9e-9144-d6a7d89517a7" containerName="maas-api" containerID="cri-o://a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6" gracePeriod=30
Sep 09 13:11:00.844486 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:00.844423    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:11:00.911059 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:00.911021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svg2k\" (UniqueName: \"kubernetes.io/projected/71b22cbf-2367-4b9e-9144-d6a7d89517a7-kube-api-access-svg2k\") pod \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") "
Sep 09 13:11:00.911253 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:00.911079    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-metrics-certs\") pod \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") "
Sep 09 13:11:00.911253 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:00.911177    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-maas-api-tls\") pod \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") "
Sep 09 13:11:00.911253 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:00.911205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b22cbf-2367-4b9e-9144-d6a7d89517a7-openshift-service-ca\") pod \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\" (UID: \"71b22cbf-2367-4b9e-9144-d6a7d89517a7\") "
Sep 09 13:11:00.912680 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:00.912641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b22cbf-2367-4b9e-9144-d6a7d89517a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71b22cbf-2367-4b9e-9144-d6a7d89517a7" (UID: "71b22cbf-2367-4b9e-9144-d6a7d89517a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:11:00.913974 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:00.913933    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b22cbf-2367-4b9e-9144-d6a7d89517a7-kube-api-access-svg2k" (OuterVolumeSpecName: "kube-api-access-svg2k") pod "71b22cbf-2367-4b9e-9144-d6a7d89517a7" (UID: "71b22cbf-2367-4b9e-9144-d6a7d89517a7"). InnerVolumeSpecName "kube-api-access-svg2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:11:00.914504 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:00.914477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71b22cbf-2367-4b9e-9144-d6a7d89517a7" (UID: "71b22cbf-2367-4b9e-9144-d6a7d89517a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:11:00.914504 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:00.914493    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71b22cbf-2367-4b9e-9144-d6a7d89517a7" (UID: "71b22cbf-2367-4b9e-9144-d6a7d89517a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:11:01.012939 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.012898    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-maas-api-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:11:01.012939 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.012932    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b22cbf-2367-4b9e-9144-d6a7d89517a7-openshift-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:11:01.013142 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.012949    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svg2k\" (UniqueName: \"kubernetes.io/projected/71b22cbf-2367-4b9e-9144-d6a7d89517a7-kube-api-access-svg2k\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:11:01.013142 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.012961    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b22cbf-2367-4b9e-9144-d6a7d89517a7-metrics-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:11:01.259398 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.259300    2580 generic.go:356] "Generic (PLEG): container finished" podID="71b22cbf-2367-4b9e-9144-d6a7d89517a7" containerID="a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6" exitCode=0
Sep 09 13:11:01.259398 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.259349    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"
Sep 09 13:11:01.259398 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.259386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2" event={"ID":"71b22cbf-2367-4b9e-9144-d6a7d89517a7","Type":"ContainerDied","Data":"a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6"}
Sep 09 13:11:01.259650 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.259440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2" event={"ID":"71b22cbf-2367-4b9e-9144-d6a7d89517a7","Type":"ContainerDied","Data":"f2766cc6c5f81754d4088a5075b7af87b2caa6af163f74d8a55d04631d361ac3"}
Sep 09 13:11:01.259650 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.259466    2580 scope.go:117] "RemoveContainer" containerID="a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6"
Sep 09 13:11:01.271359 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.271334    2580 scope.go:117] "RemoveContainer" containerID="a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6"
Sep 09 13:11:01.271675 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:11:01.271652    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6\": container with ID starting with a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6 not found: ID does not exist" containerID="a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6"
Sep 09 13:11:01.271780 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.271706    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6"} err="failed to get container status \"a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6\": rpc error: code = NotFound desc = could not find container \"a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6\": container with ID starting with a0f72296f0631e2ed8139d2cfd0541c0730ccb4b90fee74c420766e89a8273b6 not found: ID does not exist"
Sep 09 13:11:01.285971 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.285947    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"]
Sep 09 13:11:01.289388 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.289366    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-597fffff-6b5c86f4f5-68dl2"]
Sep 09 13:11:01.592407 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:01.592366    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b22cbf-2367-4b9e-9144-d6a7d89517a7" path="/var/lib/kubelet/pods/71b22cbf-2367-4b9e-9144-d6a7d89517a7/volumes"
Sep 09 13:11:03.264015 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:03.263977    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:11:19.460768 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:19.460726    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:11:34.467176 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:34.467139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:11:35.846004 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:35.845963    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"]
Sep 09 13:11:37.725442 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:37.725406    2580 scope.go:117] "RemoveContainer" containerID="ed36de413e7cf317162ae663b6208ef1521510553e4fbe39f6da00bc5f82e0ff"
Sep 09 13:11:37.736923 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:37.736897    2580 scope.go:117] "RemoveContainer" containerID="da2ca6ae271d44d903fabd8c7611b2187871598fe1524eda1d746000d4a9a394"
Sep 09 13:11:37.745173 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:37.745146    2580 scope.go:117] "RemoveContainer" containerID="6dc0021d538580adfac856d23be276a8c971f62034834d6293eb6fef20be06e6"
Sep 09 13:11:45.167740 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:45.167700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:11:50.867062 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:50.867012    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm" podUID="42683abf-51bd-4908-8906-b7a055b83e45" containerName="main" containerID="cri-o://79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60" gracePeriod=45
Sep 09 13:11:51.127486 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.127419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:11:51.222750 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.222701    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tw59\" (UniqueName: \"kubernetes.io/projected/42683abf-51bd-4908-8906-b7a055b83e45-kube-api-access-8tw59\") pod \"42683abf-51bd-4908-8906-b7a055b83e45\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") "
Sep 09 13:11:51.222971 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.222774    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-dshm\") pod \"42683abf-51bd-4908-8906-b7a055b83e45\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") "
Sep 09 13:11:51.222971 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.222866    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-model-cache\") pod \"42683abf-51bd-4908-8906-b7a055b83e45\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") "
Sep 09 13:11:51.222971 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.222887    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-tmp-dir\") pod \"42683abf-51bd-4908-8906-b7a055b83e45\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") "
Sep 09 13:11:51.222971 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.222907    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-home\") pod \"42683abf-51bd-4908-8906-b7a055b83e45\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") "
Sep 09 13:11:51.222971 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.222949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42683abf-51bd-4908-8906-b7a055b83e45-tls-certs\") pod \"42683abf-51bd-4908-8906-b7a055b83e45\" (UID: \"42683abf-51bd-4908-8906-b7a055b83e45\") "
Sep 09 13:11:51.223233 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.223173    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-model-cache" (OuterVolumeSpecName: "model-cache") pod "42683abf-51bd-4908-8906-b7a055b83e45" (UID: "42683abf-51bd-4908-8906-b7a055b83e45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:11:51.223233 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.223163    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42683abf-51bd-4908-8906-b7a055b83e45" (UID: "42683abf-51bd-4908-8906-b7a055b83e45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:11:51.223233 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.223195    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-home" (OuterVolumeSpecName: "home") pod "42683abf-51bd-4908-8906-b7a055b83e45" (UID: "42683abf-51bd-4908-8906-b7a055b83e45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:11:51.223390 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.223326    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-model-cache\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:11:51.223390 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.223339    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-tmp-dir\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:11:51.223390 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.223348    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-home\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:11:51.225287 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.225251    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42683abf-51bd-4908-8906-b7a055b83e45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42683abf-51bd-4908-8906-b7a055b83e45" (UID: "42683abf-51bd-4908-8906-b7a055b83e45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:11:51.225407 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.225300    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-dshm" (OuterVolumeSpecName: "dshm") pod "42683abf-51bd-4908-8906-b7a055b83e45" (UID: "42683abf-51bd-4908-8906-b7a055b83e45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:11:51.225407 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.225342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42683abf-51bd-4908-8906-b7a055b83e45-kube-api-access-8tw59" (OuterVolumeSpecName: "kube-api-access-8tw59") pod "42683abf-51bd-4908-8906-b7a055b83e45" (UID: "42683abf-51bd-4908-8906-b7a055b83e45"). InnerVolumeSpecName "kube-api-access-8tw59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:11:51.324555 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.324500    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tw59\" (UniqueName: \"kubernetes.io/projected/42683abf-51bd-4908-8906-b7a055b83e45-kube-api-access-8tw59\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:11:51.324555 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.324545    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42683abf-51bd-4908-8906-b7a055b83e45-dshm\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:11:51.324555 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.324559    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42683abf-51bd-4908-8906-b7a055b83e45-tls-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:11:51.539630 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.539537    2580 generic.go:356] "Generic (PLEG): container finished" podID="42683abf-51bd-4908-8906-b7a055b83e45" containerID="79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60" exitCode=0
Sep 09 13:11:51.539630 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.539580    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"
Sep 09 13:11:51.539812 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.539627    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm" event={"ID":"42683abf-51bd-4908-8906-b7a055b83e45","Type":"ContainerDied","Data":"79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60"}
Sep 09 13:11:51.539812 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.539673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm" event={"ID":"42683abf-51bd-4908-8906-b7a055b83e45","Type":"ContainerDied","Data":"665d96a51ce690caba9cc0663c0163dd8f9ad3d176e3570001086c552b2c5a1b"}
Sep 09 13:11:51.539812 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.539695    2580 scope.go:117] "RemoveContainer" containerID="79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60"
Sep 09 13:11:51.549181 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.549162    2580 scope.go:117] "RemoveContainer" containerID="79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60"
Sep 09 13:11:51.549427 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:11:51.549409    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60\": container with ID starting with 79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60 not found: ID does not exist" containerID="79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60"
Sep 09 13:11:51.549479 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.549436    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60"} err="failed to get container status \"79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60\": rpc error: code = NotFound desc = could not find container \"79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60\": container with ID starting with 79176260485ac404554f48550d9efc2501ccc5c9ed59a270b19f7a2c3ceb2b60 not found: ID does not exist"
Sep 09 13:11:51.560122 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.560100    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"]
Sep 09 13:11:51.565250 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.565230    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-650a99/auth-test-model-650a99-kserve-f6568db6c-4mxkm"]
Sep 09 13:11:51.591791 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:11:51.591764    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42683abf-51bd-4908-8906-b7a055b83e45" path="/var/lib/kubelet/pods/42683abf-51bd-4908-8906-b7a055b83e45/volumes"
Sep 09 13:12:00.372586 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:12:00.372546    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:12:12.272903 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:12:12.272867    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:12:30.663074 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:12:30.662977    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:12:45.768143 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:12:45.768102    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:13:01.369061 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:01.369024    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:13:05.765355 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.765313    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"]
Sep 09 13:13:05.765787 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.765771    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b22cbf-2367-4b9e-9144-d6a7d89517a7" containerName="maas-api"
Sep 09 13:13:05.765852 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.765789    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b22cbf-2367-4b9e-9144-d6a7d89517a7" containerName="maas-api"
Sep 09 13:13:05.765852 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.765809    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42683abf-51bd-4908-8906-b7a055b83e45" containerName="main"
Sep 09 13:13:05.765852 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.765829    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42683abf-51bd-4908-8906-b7a055b83e45" containerName="main"
Sep 09 13:13:05.765956 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.765909    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="42683abf-51bd-4908-8906-b7a055b83e45" containerName="main"
Sep 09 13:13:05.765956 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.765919    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b22cbf-2367-4b9e-9144-d6a7d89517a7" containerName="maas-api"
Sep 09 13:13:05.770636 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.770612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.773169 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.773138    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-21da34-serving-cert\""
Sep 09 13:13:05.782382 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.782354    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"]
Sep 09 13:13:05.871570 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.871533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-maas-api-tls\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.871754 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.871589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zw2v\" (UniqueName: \"kubernetes.io/projected/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-kube-api-access-6zw2v\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.871754 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.871611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-openshift-service-ca\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.871754 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.871729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-metrics-certs\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.973194 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.973155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-metrics-certs\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.973421 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.973300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-maas-api-tls\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.973421 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.973361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zw2v\" (UniqueName: \"kubernetes.io/projected/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-kube-api-access-6zw2v\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.973557 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.973479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-openshift-service-ca\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.974093 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.974070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-openshift-service-ca\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.975779 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.975761    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-metrics-certs\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.975899 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.975797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-maas-api-tls\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:05.981169 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:05.981143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zw2v\" (UniqueName: \"kubernetes.io/projected/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-kube-api-access-6zw2v\") pod \"maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:06.082901 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:06.082859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:06.354596 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:06.354566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"]
Sep 09 13:13:06.356862 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:13:06.356831    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e9a55b_4b3f_4fec_a5e2_6941f251597f.slice/crio-6ececaa1148fafb6cf0151f787bc97b87a0469f5ff8ff00fba26019048439f00 WatchSource:0}: Error finding container 6ececaa1148fafb6cf0151f787bc97b87a0469f5ff8ff00fba26019048439f00: Status 404 returned error can't find the container with id 6ececaa1148fafb6cf0151f787bc97b87a0469f5ff8ff00fba26019048439f00
Sep 09 13:13:06.358378 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:06.358359    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 13:13:06.940665 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:06.940626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m" event={"ID":"a8e9a55b-4b3f-4fec-a5e2-6941f251597f","Type":"ContainerStarted","Data":"6ececaa1148fafb6cf0151f787bc97b87a0469f5ff8ff00fba26019048439f00"}
Sep 09 13:13:07.947283 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:07.947245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m" event={"ID":"a8e9a55b-4b3f-4fec-a5e2-6941f251597f","Type":"ContainerStarted","Data":"0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538"}
Sep 09 13:13:07.947747 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:07.947305    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:07.964447 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:07.964392    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m" podStartSLOduration=2.124506491 podStartE2EDuration="2.964375978s" totalImagesPullingTime="839.869487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:13:07.963688974 +0000 UTC m=+1470.883580313" watchObservedRunningTime="2026-09-09 13:13:07.964375978 +0000 UTC m=+1470.884267378"
Sep 09 13:13:13.958275 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:13.958239    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:13:14.969157 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:14.969113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:13:29.867556 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:29.867519    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:13:35.207265 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.207235    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"]
Sep 09 13:13:35.211274 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.211256    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.213785 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.213761    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a813cb-serving-cert\""
Sep 09 13:13:35.221570 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.221541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"]
Sep 09 13:13:35.368062 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.368029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-949xx\" (UniqueName: \"kubernetes.io/projected/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-kube-api-access-949xx\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.368062 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.368067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-maas-api-tls\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.368309 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.368202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-openshift-service-ca\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.368309 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.368268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-metrics-certs\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.469679 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.469591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-metrics-certs\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.469679 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.469662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-949xx\" (UniqueName: \"kubernetes.io/projected/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-kube-api-access-949xx\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.469679 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.469683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-maas-api-tls\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.469966 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.469748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-openshift-service-ca\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.470398 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.470380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-openshift-service-ca\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.472297 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.472274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-maas-api-tls\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.472386 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.472347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-metrics-certs\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.477563 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.477545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-949xx\" (UniqueName: \"kubernetes.io/projected/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-kube-api-access-949xx\") pod \"maas-api-e2e-api-b-a813cb-866548b74d-wbkzc\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.526446 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.526411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:35.799863 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.799813    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"]
Sep 09 13:13:35.802571 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:13:35.802532    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c458f9b_2fc8_4e82_8ef7_38fb53c28d34.slice/crio-cb9fbc69911c67c62836c6dbc07778cf18ecc9bd08ad8516bc0bbf455c271ff1 WatchSource:0}: Error finding container cb9fbc69911c67c62836c6dbc07778cf18ecc9bd08ad8516bc0bbf455c271ff1: Status 404 returned error can't find the container with id cb9fbc69911c67c62836c6dbc07778cf18ecc9bd08ad8516bc0bbf455c271ff1
Sep 09 13:13:35.925008 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.924971    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"]
Sep 09 13:13:35.929977 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.929959    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:13:35.932428 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.932406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a813cb-dockercfg-6s84v\""
Sep 09 13:13:35.932428 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.932424    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a813cb\""
Sep 09 13:13:35.938208 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:35.938187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"]
Sep 09 13:13:36.075463 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:36.075421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426b55ac-e96d-4faf-b56b-d708492dd876-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht\" (UID: \"426b55ac-e96d-4faf-b56b-d708492dd876\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:13:36.075672 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:36.075574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsp2v\" (UniqueName: \"kubernetes.io/projected/426b55ac-e96d-4faf-b56b-d708492dd876-kube-api-access-qsp2v\") pod \"payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht\" (UID: \"426b55ac-e96d-4faf-b56b-d708492dd876\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:13:36.097314 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:36.097281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc" event={"ID":"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34","Type":"ContainerStarted","Data":"cb9fbc69911c67c62836c6dbc07778cf18ecc9bd08ad8516bc0bbf455c271ff1"}
Sep 09 13:13:36.176790 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:36.176761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsp2v\" (UniqueName: \"kubernetes.io/projected/426b55ac-e96d-4faf-b56b-d708492dd876-kube-api-access-qsp2v\") pod \"payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht\" (UID: \"426b55ac-e96d-4faf-b56b-d708492dd876\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:13:36.176934 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:36.176887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426b55ac-e96d-4faf-b56b-d708492dd876-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht\" (UID: \"426b55ac-e96d-4faf-b56b-d708492dd876\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:13:36.177661 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:36.177585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426b55ac-e96d-4faf-b56b-d708492dd876-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht\" (UID: \"426b55ac-e96d-4faf-b56b-d708492dd876\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:13:36.185447 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:36.185423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsp2v\" (UniqueName: \"kubernetes.io/projected/426b55ac-e96d-4faf-b56b-d708492dd876-kube-api-access-qsp2v\") pod \"payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht\" (UID: \"426b55ac-e96d-4faf-b56b-d708492dd876\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:13:36.242679 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:36.242644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:13:36.515133 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:36.515084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"]
Sep 09 13:13:36.517503 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:13:36.517473    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426b55ac_e96d_4faf_b56b_d708492dd876.slice/crio-896aa0fcd2e06be7cbd173407507e2390143fa3488fa4b74dc163e47639f1bb3 WatchSource:0}: Error finding container 896aa0fcd2e06be7cbd173407507e2390143fa3488fa4b74dc163e47639f1bb3: Status 404 returned error can't find the container with id 896aa0fcd2e06be7cbd173407507e2390143fa3488fa4b74dc163e47639f1bb3
Sep 09 13:13:37.104781 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:37.104740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc" event={"ID":"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34","Type":"ContainerStarted","Data":"3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3"}
Sep 09 13:13:37.104998 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:37.104876    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:37.106495 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:37.106469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht" event={"ID":"426b55ac-e96d-4faf-b56b-d708492dd876","Type":"ContainerStarted","Data":"3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0"}
Sep 09 13:13:37.106604 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:37.106511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht" event={"ID":"426b55ac-e96d-4faf-b56b-d708492dd876","Type":"ContainerStarted","Data":"896aa0fcd2e06be7cbd173407507e2390143fa3488fa4b74dc163e47639f1bb3"}
Sep 09 13:13:37.106604 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:37.106530    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:13:37.123650 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:37.123599    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc" podStartSLOduration=1.787264055 podStartE2EDuration="2.123583062s" totalImagesPullingTime="336.319007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:13:37.12227597 +0000 UTC m=+1500.042167302" watchObservedRunningTime="2026-09-09 13:13:37.123583062 +0000 UTC m=+1500.043474461"
Sep 09 13:13:37.140026 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:37.139979    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht" podStartSLOduration=2.139963612 podStartE2EDuration="2.139963612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:13:37.13839877 +0000 UTC m=+1500.058290122" watchObservedRunningTime="2026-09-09 13:13:37.139963612 +0000 UTC m=+1500.059854951"
Sep 09 13:13:37.789061 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:37.789030    2580 scope.go:117] "RemoveContainer" containerID="7349ad750ff7784e8484601e661624075d5f28c6249fbde0af95f1619c6b4b14"
Sep 09 13:13:37.798654 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:37.798632    2580 scope.go:117] "RemoveContainer" containerID="d827dec0c20ef1e0eeae59493b5f49b69a5a00e1f5e8c8078873464c0646a582"
Sep 09 13:13:38.188811 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:38.188783    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 13:13:38.193304 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:38.193278    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 13:13:42.374963 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:42.374930    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:13:43.118123 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:43.118085    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:13:48.113837 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:48.113794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:13:59.978401 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:13:59.978315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:14:17.980433 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:17.980399    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"]
Sep 09 13:14:17.983165 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:17.980624    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m" podUID="a8e9a55b-4b3f-4fec-a5e2-6941f251597f" containerName="maas-api" containerID="cri-o://0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538" gracePeriod=30
Sep 09 13:14:18.230546 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.230469    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:14:18.302841 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.302793    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-openshift-service-ca\") pod \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") "
Sep 09 13:14:18.302841 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.302848    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-metrics-certs\") pod \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") "
Sep 09 13:14:18.303053 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.302871    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zw2v\" (UniqueName: \"kubernetes.io/projected/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-kube-api-access-6zw2v\") pod \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") "
Sep 09 13:14:18.303053 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.302899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-maas-api-tls\") pod \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\" (UID: \"a8e9a55b-4b3f-4fec-a5e2-6941f251597f\") "
Sep 09 13:14:18.303320 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.303295    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8e9a55b-4b3f-4fec-a5e2-6941f251597f" (UID: "a8e9a55b-4b3f-4fec-a5e2-6941f251597f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:14:18.305094 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.305069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a8e9a55b-4b3f-4fec-a5e2-6941f251597f" (UID: "a8e9a55b-4b3f-4fec-a5e2-6941f251597f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:14:18.305194 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.305121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8e9a55b-4b3f-4fec-a5e2-6941f251597f" (UID: "a8e9a55b-4b3f-4fec-a5e2-6941f251597f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:14:18.305234 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.305206    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-kube-api-access-6zw2v" (OuterVolumeSpecName: "kube-api-access-6zw2v") pod "a8e9a55b-4b3f-4fec-a5e2-6941f251597f" (UID: "a8e9a55b-4b3f-4fec-a5e2-6941f251597f"). InnerVolumeSpecName "kube-api-access-6zw2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:14:18.329101 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.329068    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8e9a55b-4b3f-4fec-a5e2-6941f251597f" containerID="0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538" exitCode=0
Sep 09 13:14:18.329252 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.329109    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"
Sep 09 13:14:18.329252 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.329143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m" event={"ID":"a8e9a55b-4b3f-4fec-a5e2-6941f251597f","Type":"ContainerDied","Data":"0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538"}
Sep 09 13:14:18.329252 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.329196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m" event={"ID":"a8e9a55b-4b3f-4fec-a5e2-6941f251597f","Type":"ContainerDied","Data":"6ececaa1148fafb6cf0151f787bc97b87a0469f5ff8ff00fba26019048439f00"}
Sep 09 13:14:18.329252 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.329218    2580 scope.go:117] "RemoveContainer" containerID="0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538"
Sep 09 13:14:18.339138 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.339121    2580 scope.go:117] "RemoveContainer" containerID="0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538"
Sep 09 13:14:18.339411 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:14:18.339393    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538\": container with ID starting with 0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538 not found: ID does not exist" containerID="0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538"
Sep 09 13:14:18.339461 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.339420    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538"} err="failed to get container status \"0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538\": rpc error: code = NotFound desc = could not find container \"0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538\": container with ID starting with 0b39e40a5c7c27eca081fd14d8a4423777dbb05129055bf88b47d23df3f56538 not found: ID does not exist"
Sep 09 13:14:18.351363 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.351340    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"]
Sep 09 13:14:18.357944 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.357919    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21da34-5c66b75f58-p5h8m"]
Sep 09 13:14:18.404112 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.404075    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-maas-api-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:18.404112 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.404102    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-openshift-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:18.404112 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.404111    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-metrics-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:18.404112 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:18.404120    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zw2v\" (UniqueName: \"kubernetes.io/projected/a8e9a55b-4b3f-4fec-a5e2-6941f251597f-kube-api-access-6zw2v\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:19.592316 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:19.592283    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8e9a55b-4b3f-4fec-a5e2-6941f251597f" path="/var/lib/kubelet/pods/a8e9a55b-4b3f-4fec-a5e2-6941f251597f/volumes"
Sep 09 13:14:23.075208 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:23.075179    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:14:40.403889 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.403853    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"]
Sep 09 13:14:40.404365 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.404136    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc" podUID="0c458f9b-2fc8-4e82-8ef7-38fb53c28d34" containerName="maas-api" containerID="cri-o://3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3" gracePeriod=30
Sep 09 13:14:40.459657 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.459621    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"]
Sep 09 13:14:40.460147 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.460117    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht" podUID="426b55ac-e96d-4faf-b56b-d708492dd876" containerName="payload-pre-processing" containerID="cri-o://3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0" gracePeriod=30
Sep 09 13:14:40.722942 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.722920    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:14:40.738644 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.738620    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:14:40.832473 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.832440    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-949xx\" (UniqueName: \"kubernetes.io/projected/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-kube-api-access-949xx\") pod \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") "
Sep 09 13:14:40.832473 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.832477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-maas-api-tls\") pod \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") "
Sep 09 13:14:40.832745 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.832502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426b55ac-e96d-4faf-b56b-d708492dd876-plugins-config-volume\") pod \"426b55ac-e96d-4faf-b56b-d708492dd876\" (UID: \"426b55ac-e96d-4faf-b56b-d708492dd876\") "
Sep 09 13:14:40.832745 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.832523    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-openshift-service-ca\") pod \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") "
Sep 09 13:14:40.832745 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.832557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsp2v\" (UniqueName: \"kubernetes.io/projected/426b55ac-e96d-4faf-b56b-d708492dd876-kube-api-access-qsp2v\") pod \"426b55ac-e96d-4faf-b56b-d708492dd876\" (UID: \"426b55ac-e96d-4faf-b56b-d708492dd876\") "
Sep 09 13:14:40.832745 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.832592    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-metrics-certs\") pod \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\" (UID: \"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34\") "
Sep 09 13:14:40.833005 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.832942    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c458f9b-2fc8-4e82-8ef7-38fb53c28d34" (UID: "0c458f9b-2fc8-4e82-8ef7-38fb53c28d34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:14:40.833005 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.832979    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426b55ac-e96d-4faf-b56b-d708492dd876-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "426b55ac-e96d-4faf-b56b-d708492dd876" (UID: "426b55ac-e96d-4faf-b56b-d708492dd876"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:14:40.833142 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.833112    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426b55ac-e96d-4faf-b56b-d708492dd876-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:40.833211 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.833147    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-openshift-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:40.835063 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.835032    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-kube-api-access-949xx" (OuterVolumeSpecName: "kube-api-access-949xx") pod "0c458f9b-2fc8-4e82-8ef7-38fb53c28d34" (UID: "0c458f9b-2fc8-4e82-8ef7-38fb53c28d34"). InnerVolumeSpecName "kube-api-access-949xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:14:40.835175 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.835070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c458f9b-2fc8-4e82-8ef7-38fb53c28d34" (UID: "0c458f9b-2fc8-4e82-8ef7-38fb53c28d34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:14:40.835175 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.835095    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426b55ac-e96d-4faf-b56b-d708492dd876-kube-api-access-qsp2v" (OuterVolumeSpecName: "kube-api-access-qsp2v") pod "426b55ac-e96d-4faf-b56b-d708492dd876" (UID: "426b55ac-e96d-4faf-b56b-d708492dd876"). InnerVolumeSpecName "kube-api-access-qsp2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:14:40.835175 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.835159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0c458f9b-2fc8-4e82-8ef7-38fb53c28d34" (UID: "0c458f9b-2fc8-4e82-8ef7-38fb53c28d34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:14:40.934229 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.934188    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-949xx\" (UniqueName: \"kubernetes.io/projected/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-kube-api-access-949xx\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:40.934229 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.934222    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-maas-api-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:40.934229 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.934237    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsp2v\" (UniqueName: \"kubernetes.io/projected/426b55ac-e96d-4faf-b56b-d708492dd876-kube-api-access-qsp2v\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:40.934471 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:40.934251    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34-metrics-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:41.457150 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.457112    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c458f9b-2fc8-4e82-8ef7-38fb53c28d34" containerID="3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3" exitCode=0
Sep 09 13:14:41.457599 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.457154    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"
Sep 09 13:14:41.457599 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.457201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc" event={"ID":"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34","Type":"ContainerDied","Data":"3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3"}
Sep 09 13:14:41.457599 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.457258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc" event={"ID":"0c458f9b-2fc8-4e82-8ef7-38fb53c28d34","Type":"ContainerDied","Data":"cb9fbc69911c67c62836c6dbc07778cf18ecc9bd08ad8516bc0bbf455c271ff1"}
Sep 09 13:14:41.457599 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.457283    2580 scope.go:117] "RemoveContainer" containerID="3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3"
Sep 09 13:14:41.458673 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.458652    2580 generic.go:356] "Generic (PLEG): container finished" podID="426b55ac-e96d-4faf-b56b-d708492dd876" containerID="3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0" exitCode=0
Sep 09 13:14:41.458778 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.458688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht" event={"ID":"426b55ac-e96d-4faf-b56b-d708492dd876","Type":"ContainerDied","Data":"3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0"}
Sep 09 13:14:41.458778 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.458705    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"
Sep 09 13:14:41.458778 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.458729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht" event={"ID":"426b55ac-e96d-4faf-b56b-d708492dd876","Type":"ContainerDied","Data":"896aa0fcd2e06be7cbd173407507e2390143fa3488fa4b74dc163e47639f1bb3"}
Sep 09 13:14:41.468138 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.468117    2580 scope.go:117] "RemoveContainer" containerID="3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3"
Sep 09 13:14:41.468508 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:14:41.468444    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3\": container with ID starting with 3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3 not found: ID does not exist" containerID="3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3"
Sep 09 13:14:41.468508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.468468    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3"} err="failed to get container status \"3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3\": rpc error: code = NotFound desc = could not find container \"3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3\": container with ID starting with 3e5a42b17a94b840ad725f2c72226123b6a843b9ad36637216eb04279c3b3ef3 not found: ID does not exist"
Sep 09 13:14:41.468508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.468489    2580 scope.go:117] "RemoveContainer" containerID="3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0"
Sep 09 13:14:41.477394 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.477225    2580 scope.go:117] "RemoveContainer" containerID="3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0"
Sep 09 13:14:41.477561 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:14:41.477539    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0\": container with ID starting with 3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0 not found: ID does not exist" containerID="3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0"
Sep 09 13:14:41.477636 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.477568    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0"} err="failed to get container status \"3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0\": rpc error: code = NotFound desc = could not find container \"3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0\": container with ID starting with 3f01b8988c0ada74afea4c20f2533632e0055482ac043bed64d82e69496c25e0 not found: ID does not exist"
Sep 09 13:14:41.479497 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.479472    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"]
Sep 09 13:14:41.484613 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.484585    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813cb-866548b74d-wbkzc"]
Sep 09 13:14:41.494731 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.494709    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"]
Sep 09 13:14:41.498239 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.498213    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813cb-5694898ccd-4rfht"]
Sep 09 13:14:41.593072 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.593044    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c458f9b-2fc8-4e82-8ef7-38fb53c28d34" path="/var/lib/kubelet/pods/0c458f9b-2fc8-4e82-8ef7-38fb53c28d34/volumes"
Sep 09 13:14:41.593452 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:41.593438    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426b55ac-e96d-4faf-b56b-d708492dd876" path="/var/lib/kubelet/pods/426b55ac-e96d-4faf-b56b-d708492dd876/volumes"
Sep 09 13:14:44.867482 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:44.867445    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:14:55.175939 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.175893    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"]
Sep 09 13:14:55.176596 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.176575    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426b55ac-e96d-4faf-b56b-d708492dd876" containerName="payload-pre-processing"
Sep 09 13:14:55.176671 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.176601    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="426b55ac-e96d-4faf-b56b-d708492dd876" containerName="payload-pre-processing"
Sep 09 13:14:55.176671 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.176622    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8e9a55b-4b3f-4fec-a5e2-6941f251597f" containerName="maas-api"
Sep 09 13:14:55.176671 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.176630    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8e9a55b-4b3f-4fec-a5e2-6941f251597f" containerName="maas-api"
Sep 09 13:14:55.176671 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.176649    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c458f9b-2fc8-4e82-8ef7-38fb53c28d34" containerName="maas-api"
Sep 09 13:14:55.176671 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.176660    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c458f9b-2fc8-4e82-8ef7-38fb53c28d34" containerName="maas-api"
Sep 09 13:14:55.176983 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.176785    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c458f9b-2fc8-4e82-8ef7-38fb53c28d34" containerName="maas-api"
Sep 09 13:14:55.176983 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.176796    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8e9a55b-4b3f-4fec-a5e2-6941f251597f" containerName="maas-api"
Sep 09 13:14:55.176983 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.176806    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="426b55ac-e96d-4faf-b56b-d708492dd876" containerName="payload-pre-processing"
Sep 09 13:14:55.182465 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.182438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.184952 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.184929    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-458b0f-serving-cert\""
Sep 09 13:14:55.190200 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.190173    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"]
Sep 09 13:14:55.275903 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.275868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.275903 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.275911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98fa770d-4e44-4f31-88a1-ceed95d99d16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.276131 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.275942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-metrics-certs\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.276131 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.276073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2g84\" (UniqueName: \"kubernetes.io/projected/98fa770d-4e44-4f31-88a1-ceed95d99d16-kube-api-access-m2g84\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.377237 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.377189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.377237 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.377240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98fa770d-4e44-4f31-88a1-ceed95d99d16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.377517 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.377271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-metrics-certs\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.377517 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.377333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2g84\" (UniqueName: \"kubernetes.io/projected/98fa770d-4e44-4f31-88a1-ceed95d99d16-kube-api-access-m2g84\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.378065 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.378040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98fa770d-4e44-4f31-88a1-ceed95d99d16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.379932 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.379910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.380028 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.379956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-metrics-certs\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.385039 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.385014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2g84\" (UniqueName: \"kubernetes.io/projected/98fa770d-4e44-4f31-88a1-ceed95d99d16-kube-api-access-m2g84\") pod \"maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.495637 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.495543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:55.772119 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:55.772094    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"]
Sep 09 13:14:55.774955 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:14:55.774925    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fa770d_4e44_4f31_88a1_ceed95d99d16.slice/crio-6fca9f853eb909d9230a774f6d52d101875aabcd0fd66cb0f42f466d876ea5c0 WatchSource:0}: Error finding container 6fca9f853eb909d9230a774f6d52d101875aabcd0fd66cb0f42f466d876ea5c0: Status 404 returned error can't find the container with id 6fca9f853eb909d9230a774f6d52d101875aabcd0fd66cb0f42f466d876ea5c0
Sep 09 13:14:56.540385 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:56.540304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9" event={"ID":"98fa770d-4e44-4f31-88a1-ceed95d99d16","Type":"ContainerStarted","Data":"1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491"}
Sep 09 13:14:56.540741 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:56.540388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9" event={"ID":"98fa770d-4e44-4f31-88a1-ceed95d99d16","Type":"ContainerStarted","Data":"6fca9f853eb909d9230a774f6d52d101875aabcd0fd66cb0f42f466d876ea5c0"}
Sep 09 13:14:56.540741 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:56.540453    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:14:56.557918 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:56.557863    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9" podStartSLOduration=1.046773715 podStartE2EDuration="1.557848214s" totalImagesPullingTime="511.074499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:14:56.55717649 +0000 UTC m=+1579.477067828" watchObservedRunningTime="2026-09-09 13:14:56.557848214 +0000 UTC m=+1579.477739553"
Sep 09 13:14:58.137111 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.137078    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"]
Sep 09 13:14:58.137535 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.137350    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl" podUID="7456ae98-5004-4a87-817d-b39def859cb7" containerName="payload-processing" containerID="cri-o://848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda" gracePeriod=30
Sep 09 13:14:58.144927 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.144896    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"]
Sep 09 13:14:58.145215 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.145179    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j" podUID="23cbe1ad-69c6-4592-b104-5fec49466934" containerName="payload-pre-processing" containerID="cri-o://c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97" gracePeriod=30
Sep 09 13:14:58.434854 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.434811    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:14:58.437889 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.437869    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:14:58.510408 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.510377    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23cbe1ad-69c6-4592-b104-5fec49466934-plugins-config-volume\") pod \"23cbe1ad-69c6-4592-b104-5fec49466934\" (UID: \"23cbe1ad-69c6-4592-b104-5fec49466934\") "
Sep 09 13:14:58.510609 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.510418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4bs9\" (UniqueName: \"kubernetes.io/projected/23cbe1ad-69c6-4592-b104-5fec49466934-kube-api-access-h4bs9\") pod \"23cbe1ad-69c6-4592-b104-5fec49466934\" (UID: \"23cbe1ad-69c6-4592-b104-5fec49466934\") "
Sep 09 13:14:58.510609 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.510447    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znvzm\" (UniqueName: \"kubernetes.io/projected/7456ae98-5004-4a87-817d-b39def859cb7-kube-api-access-znvzm\") pod \"7456ae98-5004-4a87-817d-b39def859cb7\" (UID: \"7456ae98-5004-4a87-817d-b39def859cb7\") "
Sep 09 13:14:58.510609 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.510490    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7456ae98-5004-4a87-817d-b39def859cb7-plugins-config-volume\") pod \"7456ae98-5004-4a87-817d-b39def859cb7\" (UID: \"7456ae98-5004-4a87-817d-b39def859cb7\") "
Sep 09 13:14:58.510882 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.510842    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cbe1ad-69c6-4592-b104-5fec49466934-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23cbe1ad-69c6-4592-b104-5fec49466934" (UID: "23cbe1ad-69c6-4592-b104-5fec49466934"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:14:58.510970 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.510949    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7456ae98-5004-4a87-817d-b39def859cb7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7456ae98-5004-4a87-817d-b39def859cb7" (UID: "7456ae98-5004-4a87-817d-b39def859cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:14:58.512749 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.512727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cbe1ad-69c6-4592-b104-5fec49466934-kube-api-access-h4bs9" (OuterVolumeSpecName: "kube-api-access-h4bs9") pod "23cbe1ad-69c6-4592-b104-5fec49466934" (UID: "23cbe1ad-69c6-4592-b104-5fec49466934"). InnerVolumeSpecName "kube-api-access-h4bs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:14:58.512749 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.512734    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7456ae98-5004-4a87-817d-b39def859cb7-kube-api-access-znvzm" (OuterVolumeSpecName: "kube-api-access-znvzm") pod "7456ae98-5004-4a87-817d-b39def859cb7" (UID: "7456ae98-5004-4a87-817d-b39def859cb7"). InnerVolumeSpecName "kube-api-access-znvzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:14:58.551729 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.551691    2580 generic.go:356] "Generic (PLEG): container finished" podID="23cbe1ad-69c6-4592-b104-5fec49466934" containerID="c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97" exitCode=0
Sep 09 13:14:58.551903 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.551734    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"
Sep 09 13:14:58.551903 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.551779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j" event={"ID":"23cbe1ad-69c6-4592-b104-5fec49466934","Type":"ContainerDied","Data":"c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97"}
Sep 09 13:14:58.551903 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.551864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j" event={"ID":"23cbe1ad-69c6-4592-b104-5fec49466934","Type":"ContainerDied","Data":"a4a4c700d561010fc9846e51cd0d02e468c008ca199e03d8dd38d965f2594df9"}
Sep 09 13:14:58.551903 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.551889    2580 scope.go:117] "RemoveContainer" containerID="c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97"
Sep 09 13:14:58.553379 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.553355    2580 generic.go:356] "Generic (PLEG): container finished" podID="7456ae98-5004-4a87-817d-b39def859cb7" containerID="848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda" exitCode=0
Sep 09 13:14:58.553490 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.553389    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"
Sep 09 13:14:58.553490 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.553421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl" event={"ID":"7456ae98-5004-4a87-817d-b39def859cb7","Type":"ContainerDied","Data":"848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda"}
Sep 09 13:14:58.553490 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.553460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl" event={"ID":"7456ae98-5004-4a87-817d-b39def859cb7","Type":"ContainerDied","Data":"ecbab288f185f45ee72d92408f9eb4f34d912e47a5ed9674f3c106adaa146e84"}
Sep 09 13:14:58.562801 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.562782    2580 scope.go:117] "RemoveContainer" containerID="c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97"
Sep 09 13:14:58.563206 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:14:58.563185    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97\": container with ID starting with c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97 not found: ID does not exist" containerID="c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97"
Sep 09 13:14:58.563305 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.563214    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97"} err="failed to get container status \"c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97\": rpc error: code = NotFound desc = could not find container \"c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97\": container with ID starting with c639bf88272c5e86f9675eb5a9d50a80dd5964a331bbb962fc9d21022cbaee97 not found: ID does not exist"
Sep 09 13:14:58.563305 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.563232    2580 scope.go:117] "RemoveContainer" containerID="848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda"
Sep 09 13:14:58.571508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.571491    2580 scope.go:117] "RemoveContainer" containerID="848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda"
Sep 09 13:14:58.571809 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:14:58.571791    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda\": container with ID starting with 848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda not found: ID does not exist" containerID="848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda"
Sep 09 13:14:58.571913 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.571834    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda"} err="failed to get container status \"848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda\": rpc error: code = NotFound desc = could not find container \"848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda\": container with ID starting with 848179f4ee6cf10086780c51e9d47a02c8f061c5f4a5d90f8012e77755471fda not found: ID does not exist"
Sep 09 13:14:58.576294 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.576271    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"]
Sep 09 13:14:58.580526 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.580506    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0b0a1a-59ddcfbd86-7fj4j"]
Sep 09 13:14:58.591055 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.591027    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"]
Sep 09 13:14:58.594665 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.594642    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0b0a1a-dbcbd84d7-b84zl"]
Sep 09 13:14:58.611334 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.611308    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23cbe1ad-69c6-4592-b104-5fec49466934-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:58.611334 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.611333    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4bs9\" (UniqueName: \"kubernetes.io/projected/23cbe1ad-69c6-4592-b104-5fec49466934-kube-api-access-h4bs9\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:58.611493 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.611344    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znvzm\" (UniqueName: \"kubernetes.io/projected/7456ae98-5004-4a87-817d-b39def859cb7-kube-api-access-znvzm\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:58.611493 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:58.611354    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7456ae98-5004-4a87-817d-b39def859cb7-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:14:59.592082 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:59.592046    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23cbe1ad-69c6-4592-b104-5fec49466934" path="/var/lib/kubelet/pods/23cbe1ad-69c6-4592-b104-5fec49466934/volumes"
Sep 09 13:14:59.592439 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:14:59.592428    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7456ae98-5004-4a87-817d-b39def859cb7" path="/var/lib/kubelet/pods/7456ae98-5004-4a87-817d-b39def859cb7/volumes"
Sep 09 13:15:00.134505 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.134466    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz"]
Sep 09 13:15:00.134902 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.134889    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7456ae98-5004-4a87-817d-b39def859cb7" containerName="payload-processing"
Sep 09 13:15:00.134960 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.134903    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7456ae98-5004-4a87-817d-b39def859cb7" containerName="payload-processing"
Sep 09 13:15:00.134960 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.134927    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23cbe1ad-69c6-4592-b104-5fec49466934" containerName="payload-pre-processing"
Sep 09 13:15:00.134960 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.134933    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="23cbe1ad-69c6-4592-b104-5fec49466934" containerName="payload-pre-processing"
Sep 09 13:15:00.135056 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.135001    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7456ae98-5004-4a87-817d-b39def859cb7" containerName="payload-processing"
Sep 09 13:15:00.135056 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.135011    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="23cbe1ad-69c6-4592-b104-5fec49466934" containerName="payload-pre-processing"
Sep 09 13:15:00.139871 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.139848    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz"
Sep 09 13:15:00.142494 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.142468    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t5rn5\""
Sep 09 13:15:00.147337 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.147306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz"]
Sep 09 13:15:00.152842 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.152797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz"
Sep 09 13:15:00.430834 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.430789    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz"]
Sep 09 13:15:00.433323 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:15:00.433290    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b794cc_8bbf_4128_a461_5da8f4a8ab9e.slice/crio-87f70d9e3bbef2275e73fa40e990bb75db95eac706f37edb94b9af8f0765b107 WatchSource:0}: Error finding container 87f70d9e3bbef2275e73fa40e990bb75db95eac706f37edb94b9af8f0765b107: Status 404 returned error can't find the container with id 87f70d9e3bbef2275e73fa40e990bb75db95eac706f37edb94b9af8f0765b107
Sep 09 13:15:00.574209 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:00.574171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz" event={"ID":"f7b794cc-8bbf-4128-a461-5da8f4a8ab9e","Type":"ContainerStarted","Data":"87f70d9e3bbef2275e73fa40e990bb75db95eac706f37edb94b9af8f0765b107"}
Sep 09 13:15:01.581262 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:01.581224    2580 generic.go:356] "Generic (PLEG): container finished" podID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerID="6c2b27ba18aeef8195141197badb83a1625d198db3e5a5b1c8d2903a72b78cf5" exitCode=7
Sep 09 13:15:01.581688 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:01.581309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz" event={"ID":"f7b794cc-8bbf-4128-a461-5da8f4a8ab9e","Type":"ContainerDied","Data":"6c2b27ba18aeef8195141197badb83a1625d198db3e5a5b1c8d2903a72b78cf5"}
Sep 09 13:15:01.581688 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:01.581457    2580 scope.go:117] "RemoveContainer" containerID="6c2b27ba18aeef8195141197badb83a1625d198db3e5a5b1c8d2903a72b78cf5"
Sep 09 13:15:02.550841 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:02.550786    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:15:02.574182 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:02.574149    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:15:02.592054 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:02.592018    2580 generic.go:356] "Generic (PLEG): container finished" podID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerID="d4daac71db9f0fece436a383325f1b51fb061d48ffe9a6b6fa65373f29ee0263" exitCode=7
Sep 09 13:15:02.592496 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:02.592067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz" event={"ID":"f7b794cc-8bbf-4128-a461-5da8f4a8ab9e","Type":"ContainerDied","Data":"d4daac71db9f0fece436a383325f1b51fb061d48ffe9a6b6fa65373f29ee0263"}
Sep 09 13:15:02.592496 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:02.592132    2580 scope.go:117] "RemoveContainer" containerID="6c2b27ba18aeef8195141197badb83a1625d198db3e5a5b1c8d2903a72b78cf5"
Sep 09 13:15:02.592496 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:02.592314    2580 scope.go:117] "RemoveContainer" containerID="d4daac71db9f0fece436a383325f1b51fb061d48ffe9a6b6fa65373f29ee0263"
Sep 09 13:15:02.592655 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:15:02.592625    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz_odh-ai-gateway-infra(f7b794cc-8bbf-4128-a461-5da8f4a8ab9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e"
Sep 09 13:15:03.599278 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:03.599241    2580 scope.go:117] "RemoveContainer" containerID="d4daac71db9f0fece436a383325f1b51fb061d48ffe9a6b6fa65373f29ee0263"
Sep 09 13:15:03.599695 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:15:03.599482    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz_odh-ai-gateway-infra(f7b794cc-8bbf-4128-a461-5da8f4a8ab9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e"
Sep 09 13:15:16.585783 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:16.585747    2580 scope.go:117] "RemoveContainer" containerID="d4daac71db9f0fece436a383325f1b51fb061d48ffe9a6b6fa65373f29ee0263"
Sep 09 13:15:17.672899 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:17.672859    2580 generic.go:356] "Generic (PLEG): container finished" podID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerID="44d0ed77fb19a9d7b5fd612c8e078e6ad00904e62a6b6919fb84faeb3a0cf050" exitCode=0
Sep 09 13:15:17.673317 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:17.672932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz" event={"ID":"f7b794cc-8bbf-4128-a461-5da8f4a8ab9e","Type":"ContainerDied","Data":"44d0ed77fb19a9d7b5fd612c8e078e6ad00904e62a6b6919fb84faeb3a0cf050"}
Sep 09 13:15:17.673317 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:17.672994    2580 scope.go:117] "RemoveContainer" containerID="d4daac71db9f0fece436a383325f1b51fb061d48ffe9a6b6fa65373f29ee0263"
Sep 09 13:15:18.704203 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:18.704170    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz"]
Sep 09 13:15:18.825483 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:18.825455    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz"
Sep 09 13:15:19.374738 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:19.374702    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:15:19.695784 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:19.695692    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz"
Sep 09 13:15:19.695784 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:19.695735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz" event={"ID":"f7b794cc-8bbf-4128-a461-5da8f4a8ab9e","Type":"ContainerDied","Data":"87f70d9e3bbef2275e73fa40e990bb75db95eac706f37edb94b9af8f0765b107"}
Sep 09 13:15:19.696037 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:19.695847    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87f70d9e3bbef2275e73fa40e990bb75db95eac706f37edb94b9af8f0765b107"
Sep 09 13:15:19.723932 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:19.723899    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz"]
Sep 09 13:15:19.729090 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:19.729060    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-458b0f-29815995-d77wz"]
Sep 09 13:15:21.592284 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:21.592248    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" path="/var/lib/kubelet/pods/f7b794cc-8bbf-4128-a461-5da8f4a8ab9e/volumes"
Sep 09 13:15:22.387487 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:22.387449    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"]
Sep 09 13:15:22.387767 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:22.387711    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy" containerID="cri-o://1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2" gracePeriod=30
Sep 09 13:15:24.984895 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:24.984862    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"]
Sep 09 13:15:24.985357 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:24.985341    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerName="cleanup"
Sep 09 13:15:24.985357 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:24.985359    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerName="cleanup"
Sep 09 13:15:24.985457 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:24.985399    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerName="cleanup"
Sep 09 13:15:24.985457 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:24.985405    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerName="cleanup"
Sep 09 13:15:24.985527 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:24.985477    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerName="cleanup"
Sep 09 13:15:24.985527 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:24.985489    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerName="cleanup"
Sep 09 13:15:24.988901 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:24.988882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:15:24.991337 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:24.991308    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ae6936\""
Sep 09 13:15:24.991441 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:24.991368    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ae6936-dockercfg-dscd6\""
Sep 09 13:15:24.998302 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:24.998275    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"]
Sep 09 13:15:25.174991 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.174941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614fef1f-23a0-48c6-8b78-297cf37d1611-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb\" (UID: \"614fef1f-23a0-48c6-8b78-297cf37d1611\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:15:25.175189 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.175017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5qc\" (UniqueName: \"kubernetes.io/projected/614fef1f-23a0-48c6-8b78-297cf37d1611-kube-api-access-9j5qc\") pod \"payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb\" (UID: \"614fef1f-23a0-48c6-8b78-297cf37d1611\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:15:25.276593 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.276486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614fef1f-23a0-48c6-8b78-297cf37d1611-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb\" (UID: \"614fef1f-23a0-48c6-8b78-297cf37d1611\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:15:25.276593 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.276552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5qc\" (UniqueName: \"kubernetes.io/projected/614fef1f-23a0-48c6-8b78-297cf37d1611-kube-api-access-9j5qc\") pod \"payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb\" (UID: \"614fef1f-23a0-48c6-8b78-297cf37d1611\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:15:25.277301 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.277276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614fef1f-23a0-48c6-8b78-297cf37d1611-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb\" (UID: \"614fef1f-23a0-48c6-8b78-297cf37d1611\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:15:25.285990 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.285957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5qc\" (UniqueName: \"kubernetes.io/projected/614fef1f-23a0-48c6-8b78-297cf37d1611-kube-api-access-9j5qc\") pod \"payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb\" (UID: \"614fef1f-23a0-48c6-8b78-297cf37d1611\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:15:25.300854 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.300798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:15:25.578723 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.578691    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"]
Sep 09 13:15:25.580463 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:15:25.580433    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614fef1f_23a0_48c6_8b78_297cf37d1611.slice/crio-ccee795b8a55a982d5775d6c1187113e1b4d1423cf3e6edbaa57fd788d94d533 WatchSource:0}: Error finding container ccee795b8a55a982d5775d6c1187113e1b4d1423cf3e6edbaa57fd788d94d533: Status 404 returned error can't find the container with id ccee795b8a55a982d5775d6c1187113e1b4d1423cf3e6edbaa57fd788d94d533
Sep 09 13:15:25.731278 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.731237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb" event={"ID":"614fef1f-23a0-48c6-8b78-297cf37d1611","Type":"ContainerStarted","Data":"7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772"}
Sep 09 13:15:25.731590 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.731300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb" event={"ID":"614fef1f-23a0-48c6-8b78-297cf37d1611","Type":"ContainerStarted","Data":"ccee795b8a55a982d5775d6c1187113e1b4d1423cf3e6edbaa57fd788d94d533"}
Sep 09 13:15:25.731590 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.731397    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:15:25.748715 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:25.748657    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb" podStartSLOduration=1.7486413650000001 podStartE2EDuration="1.748641365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:15:25.746649896 +0000 UTC m=+1608.666541249" watchObservedRunningTime="2026-09-09 13:15:25.748641365 +0000 UTC m=+1608.668532704"
Sep 09 13:15:27.648095 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.648065    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:15:27.743661 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.743563    2580 generic.go:356] "Generic (PLEG): container finished" podID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerID="1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2" exitCode=0
Sep 09 13:15:27.743661 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.743613    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"
Sep 09 13:15:27.743661 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.743647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" event={"ID":"5be7e4c1-d44c-434f-9a3d-7e679ffd777b","Type":"ContainerDied","Data":"1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2"}
Sep 09 13:15:27.743917 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.743704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh" event={"ID":"5be7e4c1-d44c-434f-9a3d-7e679ffd777b","Type":"ContainerDied","Data":"138ff486cae3d9bdfd849f8737b6b27d66d4f3dd3a92c4eb81bfde5f82882509"}
Sep 09 13:15:27.743917 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.743736    2580 scope.go:117] "RemoveContainer" containerID="1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2"
Sep 09 13:15:27.756156 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.756134    2580 scope.go:117] "RemoveContainer" containerID="ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1"
Sep 09 13:15:27.765177 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.765152    2580 scope.go:117] "RemoveContainer" containerID="1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2"
Sep 09 13:15:27.765495 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:15:27.765472    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2\": container with ID starting with 1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2 not found: ID does not exist" containerID="1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2"
Sep 09 13:15:27.765597 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.765501    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2"} err="failed to get container status \"1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2\": rpc error: code = NotFound desc = could not find container \"1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2\": container with ID starting with 1784687d1ab28af8b38fe36f9614cd5585a2d2a8d5740a2f830b84f51e337de2 not found: ID does not exist"
Sep 09 13:15:27.765597 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.765518    2580 scope.go:117] "RemoveContainer" containerID="ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1"
Sep 09 13:15:27.765811 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:15:27.765792    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1\": container with ID starting with ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1 not found: ID does not exist" containerID="ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1"
Sep 09 13:15:27.765890 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.765846    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1"} err="failed to get container status \"ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1\": rpc error: code = NotFound desc = could not find container \"ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1\": container with ID starting with ce6c1e78d4cb231b3a3da5d4fe5484b059fff2a150b8864ed68bfaa5abb82df1 not found: ID does not exist"
Sep 09 13:15:27.805627 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.805585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-token\") pod \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") "
Sep 09 13:15:27.805801 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.805646    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-certs\") pod \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") "
Sep 09 13:15:27.805801 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.805680    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-credential-socket\") pod \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") "
Sep 09 13:15:27.805801 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.805770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-envoy\") pod \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") "
Sep 09 13:15:27.806004 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.805813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-data\") pod \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") "
Sep 09 13:15:27.806004 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.805882    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istiod-ca-cert\") pod \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") "
Sep 09 13:15:27.806004 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.805922    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwqmk\" (UniqueName: \"kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-kube-api-access-rwqmk\") pod \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") "
Sep 09 13:15:27.806156 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.805982    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5be7e4c1-d44c-434f-9a3d-7e679ffd777b" (UID: "5be7e4c1-d44c-434f-9a3d-7e679ffd777b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:15:27.806156 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.806026    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-socket\") pod \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") "
Sep 09 13:15:27.806156 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.806000    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5be7e4c1-d44c-434f-9a3d-7e679ffd777b" (UID: "5be7e4c1-d44c-434f-9a3d-7e679ffd777b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:15:27.806156 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.806079    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-podinfo\") pod \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\" (UID: \"5be7e4c1-d44c-434f-9a3d-7e679ffd777b\") "
Sep 09 13:15:27.806372 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.806161    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-data" (OuterVolumeSpecName: "istio-data") pod "5be7e4c1-d44c-434f-9a3d-7e679ffd777b" (UID: "5be7e4c1-d44c-434f-9a3d-7e679ffd777b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:15:27.806372 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.806289    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5be7e4c1-d44c-434f-9a3d-7e679ffd777b" (UID: "5be7e4c1-d44c-434f-9a3d-7e679ffd777b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:15:27.806372 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.806295    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5be7e4c1-d44c-434f-9a3d-7e679ffd777b" (UID: "5be7e4c1-d44c-434f-9a3d-7e679ffd777b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:15:27.806694 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.806463    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-credential-socket\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:15:27.806694 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.806481    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-data\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:15:27.806694 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.806494    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istiod-ca-cert\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:15:27.806694 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.806502    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-socket\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:15:27.806694 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.806510    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-workload-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:15:27.808123 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.808089    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-token" (OuterVolumeSpecName: "istio-token") pod "5be7e4c1-d44c-434f-9a3d-7e679ffd777b" (UID: "5be7e4c1-d44c-434f-9a3d-7e679ffd777b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:15:27.808258 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.808221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-kube-api-access-rwqmk" (OuterVolumeSpecName: "kube-api-access-rwqmk") pod "5be7e4c1-d44c-434f-9a3d-7e679ffd777b" (UID: "5be7e4c1-d44c-434f-9a3d-7e679ffd777b"). InnerVolumeSpecName "kube-api-access-rwqmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:15:27.808258 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.808236    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5be7e4c1-d44c-434f-9a3d-7e679ffd777b" (UID: "5be7e4c1-d44c-434f-9a3d-7e679ffd777b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:15:27.808496 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.808476    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5be7e4c1-d44c-434f-9a3d-7e679ffd777b" (UID: "5be7e4c1-d44c-434f-9a3d-7e679ffd777b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:15:27.907519 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.907477    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-podinfo\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:15:27.907519 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.907509    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-token\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:15:27.907519 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.907519    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-istio-envoy\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:15:27.907519 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:27.907527    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwqmk\" (UniqueName: \"kubernetes.io/projected/5be7e4c1-d44c-434f-9a3d-7e679ffd777b-kube-api-access-rwqmk\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:15:28.066375 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:28.066339    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"]
Sep 09 13:15:28.069938 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:28.069910    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-941953-openshift-default-6cd889845d-5jtwh"]
Sep 09 13:15:29.591983 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:29.591908    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" path="/var/lib/kubelet/pods/5be7e4c1-d44c-434f-9a3d-7e679ffd777b/volumes"
Sep 09 13:15:31.568547 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:31.568515    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:15:36.739031 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:36.738995    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:15:53.375498 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:15:53.375451    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb_614fef1f-23a0-48c6-8b78-297cf37d1611/payload-processing/0.log"
Sep 09 13:16:15.568405 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:15.568369    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:16:28.768315 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:28.768274    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:16:34.504775 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:34.504743    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb_614fef1f-23a0-48c6-8b78-297cf37d1611/payload-processing/0.log"
Sep 09 13:16:39.771976 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.771941    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"]
Sep 09 13:16:39.772612 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.772524    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy"
Sep 09 13:16:39.772612 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.772538    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy"
Sep 09 13:16:39.772612 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.772555    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy"
Sep 09 13:16:39.772612 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.772563    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy"
Sep 09 13:16:39.772612 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.772589    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerName="cleanup"
Sep 09 13:16:39.772612 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.772597    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerName="cleanup"
Sep 09 13:16:39.773039 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.772688    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy"
Sep 09 13:16:39.773039 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.772700    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be7e4c1-d44c-434f-9a3d-7e679ffd777b" containerName="istio-proxy"
Sep 09 13:16:39.773039 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.772710    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b794cc-8bbf-4128-a461-5da8f4a8ab9e" containerName="cleanup"
Sep 09 13:16:39.776140 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.776122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:39.778693 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.778671    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-964720-serving-cert\""
Sep 09 13:16:39.784035 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.783997    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"]
Sep 09 13:16:39.893445 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.893415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:39.893573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.893467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:39.893573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.893554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqqx9\" (UniqueName: \"kubernetes.io/projected/5a4ff669-ffb9-4143-84ad-b04b95e31a68-kube-api-access-pqqx9\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:39.893653 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.893587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ff669-ffb9-4143-84ad-b04b95e31a68-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:39.994762 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.994735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqqx9\" (UniqueName: \"kubernetes.io/projected/5a4ff669-ffb9-4143-84ad-b04b95e31a68-kube-api-access-pqqx9\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:39.994918 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.994767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ff669-ffb9-4143-84ad-b04b95e31a68-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:39.994918 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.994839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:39.994918 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.994888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:39.995462 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.995438    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ff669-ffb9-4143-84ad-b04b95e31a68-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:39.997569 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.997539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:39.997659 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:39.997554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:40.003755 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.003735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqqx9\" (UniqueName: \"kubernetes.io/projected/5a4ff669-ffb9-4143-84ad-b04b95e31a68-kube-api-access-pqqx9\") pod \"maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:40.089567 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.089544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:40.362777 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.362749    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"]
Sep 09 13:16:40.364738 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:16:40.364710    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4ff669_ffb9_4143_84ad_b04b95e31a68.slice/crio-45d022aa8670e1079649a46a4795cdc16011223185be8402a5d32259f2563e4b WatchSource:0}: Error finding container 45d022aa8670e1079649a46a4795cdc16011223185be8402a5d32259f2563e4b: Status 404 returned error can't find the container with id 45d022aa8670e1079649a46a4795cdc16011223185be8402a5d32259f2563e4b
Sep 09 13:16:40.400778 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.400758    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"]
Sep 09 13:16:40.405368 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.405351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:16:40.407929 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.407910    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-964720\""
Sep 09 13:16:40.408024 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.407981    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-964720-dockercfg-46g2q\""
Sep 09 13:16:40.415990 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.415968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"]
Sep 09 13:16:40.498882 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.498854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnxn\" (UniqueName: \"kubernetes.io/projected/f8dddb06-c245-407e-996d-8f7021f48375-kube-api-access-clnxn\") pod \"payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6\" (UID: \"f8dddb06-c245-407e-996d-8f7021f48375\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:16:40.499028 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.498963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8dddb06-c245-407e-996d-8f7021f48375-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6\" (UID: \"f8dddb06-c245-407e-996d-8f7021f48375\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:16:40.599678 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.599645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8dddb06-c245-407e-996d-8f7021f48375-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6\" (UID: \"f8dddb06-c245-407e-996d-8f7021f48375\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:16:40.599863 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.599735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnxn\" (UniqueName: \"kubernetes.io/projected/f8dddb06-c245-407e-996d-8f7021f48375-kube-api-access-clnxn\") pod \"payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6\" (UID: \"f8dddb06-c245-407e-996d-8f7021f48375\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:16:40.600337 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.600317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8dddb06-c245-407e-996d-8f7021f48375-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6\" (UID: \"f8dddb06-c245-407e-996d-8f7021f48375\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:16:40.607646 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.607625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnxn\" (UniqueName: \"kubernetes.io/projected/f8dddb06-c245-407e-996d-8f7021f48375-kube-api-access-clnxn\") pod \"payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6\" (UID: \"f8dddb06-c245-407e-996d-8f7021f48375\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:16:40.716893 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.716844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:16:40.987379 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:40.987260    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"]
Sep 09 13:16:40.989536 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:16:40.989508    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dddb06_c245_407e_996d_8f7021f48375.slice/crio-65dc772729e125ea39ff4ee1886f4dca8c092cd294fc3fb987dc5c0a9504109c WatchSource:0}: Error finding container 65dc772729e125ea39ff4ee1886f4dca8c092cd294fc3fb987dc5c0a9504109c: Status 404 returned error can't find the container with id 65dc772729e125ea39ff4ee1886f4dca8c092cd294fc3fb987dc5c0a9504109c
Sep 09 13:16:41.129904 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:41.129859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6" event={"ID":"f8dddb06-c245-407e-996d-8f7021f48375","Type":"ContainerStarted","Data":"64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe"}
Sep 09 13:16:41.130084 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:41.129926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6" event={"ID":"f8dddb06-c245-407e-996d-8f7021f48375","Type":"ContainerStarted","Data":"65dc772729e125ea39ff4ee1886f4dca8c092cd294fc3fb987dc5c0a9504109c"}
Sep 09 13:16:41.130084 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:41.129956    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:16:41.131213 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:41.131189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4" event={"ID":"5a4ff669-ffb9-4143-84ad-b04b95e31a68","Type":"ContainerStarted","Data":"45d022aa8670e1079649a46a4795cdc16011223185be8402a5d32259f2563e4b"}
Sep 09 13:16:41.147626 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:41.147590    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6" podStartSLOduration=1.147578605 podStartE2EDuration="1.147578605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:16:41.145862288 +0000 UTC m=+1684.065753639" watchObservedRunningTime="2026-09-09 13:16:41.147578605 +0000 UTC m=+1684.067469995"
Sep 09 13:16:42.138340 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:42.138293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4" event={"ID":"5a4ff669-ffb9-4143-84ad-b04b95e31a68","Type":"ContainerStarted","Data":"437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151"}
Sep 09 13:16:42.138794 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:42.138520    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:42.155706 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:42.155660    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4" podStartSLOduration=2.106347286 podStartE2EDuration="3.15564716s" totalImagesPullingTime="1.049299874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:16:42.15436344 +0000 UTC m=+1685.074254780" watchObservedRunningTime="2026-09-09 13:16:42.15564716 +0000 UTC m=+1685.075538499"
Sep 09 13:16:47.372669 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:47.372629    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:16:48.150111 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:48.150077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:16:52.140573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:16:52.140538    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:17:10.260054 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.259976    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"]
Sep 09 13:17:10.260556 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.260254    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4" podUID="5a4ff669-ffb9-4143-84ad-b04b95e31a68" containerName="maas-api" containerID="cri-o://437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151" gracePeriod=30
Sep 09 13:17:10.304861 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.304798    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"]
Sep 09 13:17:10.305091 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.305058    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6" podUID="f8dddb06-c245-407e-996d-8f7021f48375" containerName="payload-processing" containerID="cri-o://64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe" gracePeriod=30
Sep 09 13:17:10.556088 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.556057    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:17:10.588046 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.588014    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:17:10.697109 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.697071    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8dddb06-c245-407e-996d-8f7021f48375-plugins-config-volume\") pod \"f8dddb06-c245-407e-996d-8f7021f48375\" (UID: \"f8dddb06-c245-407e-996d-8f7021f48375\") "
Sep 09 13:17:10.697303 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.697172    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-maas-api-tls\") pod \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") "
Sep 09 13:17:10.697303 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.697264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqqx9\" (UniqueName: \"kubernetes.io/projected/5a4ff669-ffb9-4143-84ad-b04b95e31a68-kube-api-access-pqqx9\") pod \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") "
Sep 09 13:17:10.697303 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.697294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clnxn\" (UniqueName: \"kubernetes.io/projected/f8dddb06-c245-407e-996d-8f7021f48375-kube-api-access-clnxn\") pod \"f8dddb06-c245-407e-996d-8f7021f48375\" (UID: \"f8dddb06-c245-407e-996d-8f7021f48375\") "
Sep 09 13:17:10.697426 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.697330    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ff669-ffb9-4143-84ad-b04b95e31a68-openshift-service-ca\") pod \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") "
Sep 09 13:17:10.697426 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.697402    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-metrics-certs\") pod \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\" (UID: \"5a4ff669-ffb9-4143-84ad-b04b95e31a68\") "
Sep 09 13:17:10.697602 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.697570    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8dddb06-c245-407e-996d-8f7021f48375-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8dddb06-c245-407e-996d-8f7021f48375" (UID: "f8dddb06-c245-407e-996d-8f7021f48375"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:17:10.697771 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.697740    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4ff669-ffb9-4143-84ad-b04b95e31a68-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a4ff669-ffb9-4143-84ad-b04b95e31a68" (UID: "5a4ff669-ffb9-4143-84ad-b04b95e31a68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:17:10.698221 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.698176    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8dddb06-c245-407e-996d-8f7021f48375-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:17:10.698221 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.698203    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ff669-ffb9-4143-84ad-b04b95e31a68-openshift-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:17:10.699658 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.699632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a4ff669-ffb9-4143-84ad-b04b95e31a68" (UID: "5a4ff669-ffb9-4143-84ad-b04b95e31a68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:17:10.699865 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.699811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dddb06-c245-407e-996d-8f7021f48375-kube-api-access-clnxn" (OuterVolumeSpecName: "kube-api-access-clnxn") pod "f8dddb06-c245-407e-996d-8f7021f48375" (UID: "f8dddb06-c245-407e-996d-8f7021f48375"). InnerVolumeSpecName "kube-api-access-clnxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:17:10.700259 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.700239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5a4ff669-ffb9-4143-84ad-b04b95e31a68" (UID: "5a4ff669-ffb9-4143-84ad-b04b95e31a68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:17:10.700352 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.700336    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4ff669-ffb9-4143-84ad-b04b95e31a68-kube-api-access-pqqx9" (OuterVolumeSpecName: "kube-api-access-pqqx9") pod "5a4ff669-ffb9-4143-84ad-b04b95e31a68" (UID: "5a4ff669-ffb9-4143-84ad-b04b95e31a68"). InnerVolumeSpecName "kube-api-access-pqqx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:17:10.799036 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.798948    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-metrics-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:17:10.799036 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.798978    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4ff669-ffb9-4143-84ad-b04b95e31a68-maas-api-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:17:10.799036 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.798987    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqqx9\" (UniqueName: \"kubernetes.io/projected/5a4ff669-ffb9-4143-84ad-b04b95e31a68-kube-api-access-pqqx9\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:17:10.799036 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:10.798996    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clnxn\" (UniqueName: \"kubernetes.io/projected/f8dddb06-c245-407e-996d-8f7021f48375-kube-api-access-clnxn\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:17:11.297599 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.297562    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8dddb06-c245-407e-996d-8f7021f48375" containerID="64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe" exitCode=0
Sep 09 13:17:11.298115 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.297608    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"
Sep 09 13:17:11.298115 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.297649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6" event={"ID":"f8dddb06-c245-407e-996d-8f7021f48375","Type":"ContainerDied","Data":"64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe"}
Sep 09 13:17:11.298115 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.297715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6" event={"ID":"f8dddb06-c245-407e-996d-8f7021f48375","Type":"ContainerDied","Data":"65dc772729e125ea39ff4ee1886f4dca8c092cd294fc3fb987dc5c0a9504109c"}
Sep 09 13:17:11.298115 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.297753    2580 scope.go:117] "RemoveContainer" containerID="64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe"
Sep 09 13:17:11.299215 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.299191    2580 generic.go:356] "Generic (PLEG): container finished" podID="5a4ff669-ffb9-4143-84ad-b04b95e31a68" containerID="437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151" exitCode=0
Sep 09 13:17:11.299275 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.299229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4" event={"ID":"5a4ff669-ffb9-4143-84ad-b04b95e31a68","Type":"ContainerDied","Data":"437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151"}
Sep 09 13:17:11.299275 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.299242    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"
Sep 09 13:17:11.299361 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.299273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4" event={"ID":"5a4ff669-ffb9-4143-84ad-b04b95e31a68","Type":"ContainerDied","Data":"45d022aa8670e1079649a46a4795cdc16011223185be8402a5d32259f2563e4b"}
Sep 09 13:17:11.307299 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.307280    2580 scope.go:117] "RemoveContainer" containerID="64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe"
Sep 09 13:17:11.307668 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:17:11.307642    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe\": container with ID starting with 64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe not found: ID does not exist" containerID="64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe"
Sep 09 13:17:11.307762 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.307679    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe"} err="failed to get container status \"64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe\": rpc error: code = NotFound desc = could not find container \"64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe\": container with ID starting with 64aa543007ba7988ce62ef9ef24da88c71dc28a8631a9cf1e5cb436080d2b1fe not found: ID does not exist"
Sep 09 13:17:11.307762 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.307705    2580 scope.go:117] "RemoveContainer" containerID="437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151"
Sep 09 13:17:11.317351 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.317335    2580 scope.go:117] "RemoveContainer" containerID="437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151"
Sep 09 13:17:11.317624 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:17:11.317605    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151\": container with ID starting with 437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151 not found: ID does not exist" containerID="437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151"
Sep 09 13:17:11.317682 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.317629    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151"} err="failed to get container status \"437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151\": rpc error: code = NotFound desc = could not find container \"437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151\": container with ID starting with 437c338d256c611b85745320e4a987e5674b5c37f3480ea52a9e975d9ee12151 not found: ID does not exist"
Sep 09 13:17:11.336889 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.336850    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"]
Sep 09 13:17:11.338960 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.338939    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-964720-7dff8dcb7f-9ljq6"]
Sep 09 13:17:11.358339 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.358305    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"]
Sep 09 13:17:11.360151 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.360133    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-964720-59f5fc4c77-rg2z4"]
Sep 09 13:17:11.592338 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.592300    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4ff669-ffb9-4143-84ad-b04b95e31a68" path="/var/lib/kubelet/pods/5a4ff669-ffb9-4143-84ad-b04b95e31a68/volumes"
Sep 09 13:17:11.592708 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:11.592694    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8dddb06-c245-407e-996d-8f7021f48375" path="/var/lib/kubelet/pods/f8dddb06-c245-407e-996d-8f7021f48375/volumes"
Sep 09 13:17:14.668644 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:14.668611    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:17:33.670084 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:33.670047    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:17:35.999952 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:35.999909    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"]
Sep 09 13:17:36.000427 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.000166    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9" podUID="98fa770d-4e44-4f31-88a1-ceed95d99d16" containerName="maas-api" containerID="cri-o://1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491" gracePeriod=30
Sep 09 13:17:36.251088 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.251025    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:17:36.346971 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.346944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-metrics-certs\") pod \"98fa770d-4e44-4f31-88a1-ceed95d99d16\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") "
Sep 09 13:17:36.346971 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.346976    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-maas-api-tls\") pod \"98fa770d-4e44-4f31-88a1-ceed95d99d16\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") "
Sep 09 13:17:36.347180 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.347012    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2g84\" (UniqueName: \"kubernetes.io/projected/98fa770d-4e44-4f31-88a1-ceed95d99d16-kube-api-access-m2g84\") pod \"98fa770d-4e44-4f31-88a1-ceed95d99d16\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") "
Sep 09 13:17:36.347180 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.347124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98fa770d-4e44-4f31-88a1-ceed95d99d16-openshift-service-ca\") pod \"98fa770d-4e44-4f31-88a1-ceed95d99d16\" (UID: \"98fa770d-4e44-4f31-88a1-ceed95d99d16\") "
Sep 09 13:17:36.347517 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.347485    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fa770d-4e44-4f31-88a1-ceed95d99d16-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98fa770d-4e44-4f31-88a1-ceed95d99d16" (UID: "98fa770d-4e44-4f31-88a1-ceed95d99d16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:17:36.349390 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.349367    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "98fa770d-4e44-4f31-88a1-ceed95d99d16" (UID: "98fa770d-4e44-4f31-88a1-ceed95d99d16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:17:36.349500 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.349415    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fa770d-4e44-4f31-88a1-ceed95d99d16-kube-api-access-m2g84" (OuterVolumeSpecName: "kube-api-access-m2g84") pod "98fa770d-4e44-4f31-88a1-ceed95d99d16" (UID: "98fa770d-4e44-4f31-88a1-ceed95d99d16"). InnerVolumeSpecName "kube-api-access-m2g84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:17:36.349500 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.349444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98fa770d-4e44-4f31-88a1-ceed95d99d16" (UID: "98fa770d-4e44-4f31-88a1-ceed95d99d16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:17:36.429363 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.429327    2580 generic.go:356] "Generic (PLEG): container finished" podID="98fa770d-4e44-4f31-88a1-ceed95d99d16" containerID="1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491" exitCode=0
Sep 09 13:17:36.429508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.429376    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"
Sep 09 13:17:36.429508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.429372    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9" event={"ID":"98fa770d-4e44-4f31-88a1-ceed95d99d16","Type":"ContainerDied","Data":"1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491"}
Sep 09 13:17:36.429508 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.429499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9" event={"ID":"98fa770d-4e44-4f31-88a1-ceed95d99d16","Type":"ContainerDied","Data":"6fca9f853eb909d9230a774f6d52d101875aabcd0fd66cb0f42f466d876ea5c0"}
Sep 09 13:17:36.429631 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.429528    2580 scope.go:117] "RemoveContainer" containerID="1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491"
Sep 09 13:17:36.439588 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.439568    2580 scope.go:117] "RemoveContainer" containerID="1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491"
Sep 09 13:17:36.439883 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:17:36.439864    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491\": container with ID starting with 1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491 not found: ID does not exist" containerID="1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491"
Sep 09 13:17:36.439940 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.439893    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491"} err="failed to get container status \"1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491\": rpc error: code = NotFound desc = could not find container \"1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491\": container with ID starting with 1e002f3643ba1819620a8f84ec4974528b4fbc0f552df3635f4809ea5c5af491 not found: ID does not exist"
Sep 09 13:17:36.448648 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.448627    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98fa770d-4e44-4f31-88a1-ceed95d99d16-openshift-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:17:36.448648 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.448650    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-metrics-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:17:36.448780 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.448662    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98fa770d-4e44-4f31-88a1-ceed95d99d16-maas-api-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:17:36.448780 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.448670    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2g84\" (UniqueName: \"kubernetes.io/projected/98fa770d-4e44-4f31-88a1-ceed95d99d16-kube-api-access-m2g84\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:17:36.450581 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.450560    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"]
Sep 09 13:17:36.454139 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:36.454115    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-458b0f-784dffbf95-sz2l9"]
Sep 09 13:17:37.593067 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:37.593038    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98fa770d-4e44-4f31-88a1-ceed95d99d16" path="/var/lib/kubelet/pods/98fa770d-4e44-4f31-88a1-ceed95d99d16/volumes"
Sep 09 13:17:46.463927 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:17:46.463884    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:18:35.143009 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.142929    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"]
Sep 09 13:18:35.143566 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.143142    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb" podUID="614fef1f-23a0-48c6-8b78-297cf37d1611" containerName="payload-processing" containerID="cri-o://7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772" gracePeriod=30
Sep 09 13:18:35.420339 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.420304    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:18:35.524955 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.524919    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j5qc\" (UniqueName: \"kubernetes.io/projected/614fef1f-23a0-48c6-8b78-297cf37d1611-kube-api-access-9j5qc\") pod \"614fef1f-23a0-48c6-8b78-297cf37d1611\" (UID: \"614fef1f-23a0-48c6-8b78-297cf37d1611\") "
Sep 09 13:18:35.525180 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.525022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614fef1f-23a0-48c6-8b78-297cf37d1611-plugins-config-volume\") pod \"614fef1f-23a0-48c6-8b78-297cf37d1611\" (UID: \"614fef1f-23a0-48c6-8b78-297cf37d1611\") "
Sep 09 13:18:35.525353 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.525332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614fef1f-23a0-48c6-8b78-297cf37d1611-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "614fef1f-23a0-48c6-8b78-297cf37d1611" (UID: "614fef1f-23a0-48c6-8b78-297cf37d1611"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:18:35.527302 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.527282    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614fef1f-23a0-48c6-8b78-297cf37d1611-kube-api-access-9j5qc" (OuterVolumeSpecName: "kube-api-access-9j5qc") pod "614fef1f-23a0-48c6-8b78-297cf37d1611" (UID: "614fef1f-23a0-48c6-8b78-297cf37d1611"). InnerVolumeSpecName "kube-api-access-9j5qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:18:35.626778 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.626748    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5qc\" (UniqueName: \"kubernetes.io/projected/614fef1f-23a0-48c6-8b78-297cf37d1611-kube-api-access-9j5qc\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:18:35.626778 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.626776    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614fef1f-23a0-48c6-8b78-297cf37d1611-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:18:35.739290 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.739252    2580 generic.go:356] "Generic (PLEG): container finished" podID="614fef1f-23a0-48c6-8b78-297cf37d1611" containerID="7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772" exitCode=0
Sep 09 13:18:35.739438 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.739317    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"
Sep 09 13:18:35.739438 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.739323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb" event={"ID":"614fef1f-23a0-48c6-8b78-297cf37d1611","Type":"ContainerDied","Data":"7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772"}
Sep 09 13:18:35.739438 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.739383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb" event={"ID":"614fef1f-23a0-48c6-8b78-297cf37d1611","Type":"ContainerDied","Data":"ccee795b8a55a982d5775d6c1187113e1b4d1423cf3e6edbaa57fd788d94d533"}
Sep 09 13:18:35.739438 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.739409    2580 scope.go:117] "RemoveContainer" containerID="7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772"
Sep 09 13:18:35.750245 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.750223    2580 scope.go:117] "RemoveContainer" containerID="7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772"
Sep 09 13:18:35.750601 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:18:35.750573    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772\": container with ID starting with 7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772 not found: ID does not exist" containerID="7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772"
Sep 09 13:18:35.750696 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.750614    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772"} err="failed to get container status \"7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772\": rpc error: code = NotFound desc = could not find container \"7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772\": container with ID starting with 7ce6fa7d80532ff8b2f034682cc7bc7210b4027bb82bc6d08b1065eca3690772 not found: ID does not exist"
Sep 09 13:18:35.756927 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.756869    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"]
Sep 09 13:18:35.759202 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:35.759179    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae6936-578fd8c8c9-f7kjb"]
Sep 09 13:18:37.592197 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:37.592168    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="614fef1f-23a0-48c6-8b78-297cf37d1611" path="/var/lib/kubelet/pods/614fef1f-23a0-48c6-8b78-297cf37d1611/volumes"
Sep 09 13:18:38.221608 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:38.221524    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 13:18:38.228799 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:38.228775    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 13:18:38.463861 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:38.463814    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:18:52.739175 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:52.739141    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"]
Sep 09 13:18:52.739759 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:52.739358    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9" podUID="be0a2021-5883-44d5-b45f-a2d5af66f873" containerName="payload-processing" containerID="cri-o://cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2" gracePeriod=30
Sep 09 13:18:52.996199 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:52.996142    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:18:53.084848 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.084803    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5jx7\" (UniqueName: \"kubernetes.io/projected/be0a2021-5883-44d5-b45f-a2d5af66f873-kube-api-access-g5jx7\") pod \"be0a2021-5883-44d5-b45f-a2d5af66f873\" (UID: \"be0a2021-5883-44d5-b45f-a2d5af66f873\") "
Sep 09 13:18:53.085020 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.084860    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0a2021-5883-44d5-b45f-a2d5af66f873-plugins-config-volume\") pod \"be0a2021-5883-44d5-b45f-a2d5af66f873\" (UID: \"be0a2021-5883-44d5-b45f-a2d5af66f873\") "
Sep 09 13:18:53.085266 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.085235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0a2021-5883-44d5-b45f-a2d5af66f873-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be0a2021-5883-44d5-b45f-a2d5af66f873" (UID: "be0a2021-5883-44d5-b45f-a2d5af66f873"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:18:53.087255 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.087233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0a2021-5883-44d5-b45f-a2d5af66f873-kube-api-access-g5jx7" (OuterVolumeSpecName: "kube-api-access-g5jx7") pod "be0a2021-5883-44d5-b45f-a2d5af66f873" (UID: "be0a2021-5883-44d5-b45f-a2d5af66f873"). InnerVolumeSpecName "kube-api-access-g5jx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:18:53.186257 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.186227    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5jx7\" (UniqueName: \"kubernetes.io/projected/be0a2021-5883-44d5-b45f-a2d5af66f873-kube-api-access-g5jx7\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:18:53.186257 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.186251    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0a2021-5883-44d5-b45f-a2d5af66f873-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:18:53.833924 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.833891    2580 generic.go:356] "Generic (PLEG): container finished" podID="be0a2021-5883-44d5-b45f-a2d5af66f873" containerID="cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2" exitCode=0
Sep 09 13:18:53.834297 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.833942    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"
Sep 09 13:18:53.834297 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.833947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9" event={"ID":"be0a2021-5883-44d5-b45f-a2d5af66f873","Type":"ContainerDied","Data":"cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2"}
Sep 09 13:18:53.834297 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.834051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9" event={"ID":"be0a2021-5883-44d5-b45f-a2d5af66f873","Type":"ContainerDied","Data":"3e1406786306520b0982e29437f768afaec6d90cd9ec3f6dba9120ba06f588bb"}
Sep 09 13:18:53.834297 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.834075    2580 scope.go:117] "RemoveContainer" containerID="cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2"
Sep 09 13:18:53.842877 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.842857    2580 scope.go:117] "RemoveContainer" containerID="cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2"
Sep 09 13:18:53.843146 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:18:53.843126    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2\": container with ID starting with cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2 not found: ID does not exist" containerID="cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2"
Sep 09 13:18:53.843210 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.843158    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2"} err="failed to get container status \"cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2\": rpc error: code = NotFound desc = could not find container \"cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2\": container with ID starting with cee10a94f84a2d68b3c4d49f69e85140baf4890fc07463f3bd79d8a8bef1d7d2 not found: ID does not exist"
Sep 09 13:18:53.849848 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.849772    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"]
Sep 09 13:18:53.855189 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:53.855170    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ada1ae-7f4c468c9c-rt9d9"]
Sep 09 13:18:55.591944 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:55.591909    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be0a2021-5883-44d5-b45f-a2d5af66f873" path="/var/lib/kubelet/pods/be0a2021-5883-44d5-b45f-a2d5af66f873/volumes"
Sep 09 13:18:55.665539 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:18:55.665504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:19:10.165676 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.165637    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"]
Sep 09 13:19:10.166211 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.165855    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x" podUID="388cde2e-18cb-44cb-b4c4-7e3726160f01" containerName="maas-api" containerID="cri-o://2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986" gracePeriod=30
Sep 09 13:19:10.219153 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.219123    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"]
Sep 09 13:19:10.219374 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.219351    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x" podUID="af4a1b42-c978-4957-96b9-efa47269c720" containerName="payload-processing" containerID="cri-o://f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5" gracePeriod=30
Sep 09 13:19:10.476543 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.476522    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:19:10.479422 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.479401    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:19:10.650455 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.650426    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n78ww\" (UniqueName: \"kubernetes.io/projected/388cde2e-18cb-44cb-b4c4-7e3726160f01-kube-api-access-n78ww\") pod \"388cde2e-18cb-44cb-b4c4-7e3726160f01\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") "
Sep 09 13:19:10.650606 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.650464    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfl4d\" (UniqueName: \"kubernetes.io/projected/af4a1b42-c978-4957-96b9-efa47269c720-kube-api-access-pfl4d\") pod \"af4a1b42-c978-4957-96b9-efa47269c720\" (UID: \"af4a1b42-c978-4957-96b9-efa47269c720\") "
Sep 09 13:19:10.650606 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.650500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-metrics-certs\") pod \"388cde2e-18cb-44cb-b4c4-7e3726160f01\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") "
Sep 09 13:19:10.650606 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.650521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-maas-api-tls\") pod \"388cde2e-18cb-44cb-b4c4-7e3726160f01\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") "
Sep 09 13:19:10.650751 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.650639    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4a1b42-c978-4957-96b9-efa47269c720-plugins-config-volume\") pod \"af4a1b42-c978-4957-96b9-efa47269c720\" (UID: \"af4a1b42-c978-4957-96b9-efa47269c720\") "
Sep 09 13:19:10.650751 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.650738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388cde2e-18cb-44cb-b4c4-7e3726160f01-openshift-service-ca\") pod \"388cde2e-18cb-44cb-b4c4-7e3726160f01\" (UID: \"388cde2e-18cb-44cb-b4c4-7e3726160f01\") "
Sep 09 13:19:10.651392 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.651124    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4a1b42-c978-4957-96b9-efa47269c720-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af4a1b42-c978-4957-96b9-efa47269c720" (UID: "af4a1b42-c978-4957-96b9-efa47269c720"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:19:10.651392 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.651187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388cde2e-18cb-44cb-b4c4-7e3726160f01-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "388cde2e-18cb-44cb-b4c4-7e3726160f01" (UID: "388cde2e-18cb-44cb-b4c4-7e3726160f01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:19:10.652855 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.652814    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4a1b42-c978-4957-96b9-efa47269c720-kube-api-access-pfl4d" (OuterVolumeSpecName: "kube-api-access-pfl4d") pod "af4a1b42-c978-4957-96b9-efa47269c720" (UID: "af4a1b42-c978-4957-96b9-efa47269c720"). InnerVolumeSpecName "kube-api-access-pfl4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:19:10.652950 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.652871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "388cde2e-18cb-44cb-b4c4-7e3726160f01" (UID: "388cde2e-18cb-44cb-b4c4-7e3726160f01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:19:10.652950 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.652889    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388cde2e-18cb-44cb-b4c4-7e3726160f01-kube-api-access-n78ww" (OuterVolumeSpecName: "kube-api-access-n78ww") pod "388cde2e-18cb-44cb-b4c4-7e3726160f01" (UID: "388cde2e-18cb-44cb-b4c4-7e3726160f01"). InnerVolumeSpecName "kube-api-access-n78ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:19:10.652950 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.652896    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "388cde2e-18cb-44cb-b4c4-7e3726160f01" (UID: "388cde2e-18cb-44cb-b4c4-7e3726160f01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:19:10.752308 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.752243    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388cde2e-18cb-44cb-b4c4-7e3726160f01-openshift-service-ca\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:19:10.752308 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.752266    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n78ww\" (UniqueName: \"kubernetes.io/projected/388cde2e-18cb-44cb-b4c4-7e3726160f01-kube-api-access-n78ww\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:19:10.752308 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.752275    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfl4d\" (UniqueName: \"kubernetes.io/projected/af4a1b42-c978-4957-96b9-efa47269c720-kube-api-access-pfl4d\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:19:10.752308 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.752284    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-metrics-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:19:10.752308 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.752293    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388cde2e-18cb-44cb-b4c4-7e3726160f01-maas-api-tls\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:19:10.752308 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.752300    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4a1b42-c978-4957-96b9-efa47269c720-plugins-config-volume\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:19:10.918312 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.918279    2580 generic.go:356] "Generic (PLEG): container finished" podID="af4a1b42-c978-4957-96b9-efa47269c720" containerID="f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5" exitCode=0
Sep 09 13:19:10.918469 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.918325    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"
Sep 09 13:19:10.918469 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.918366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x" event={"ID":"af4a1b42-c978-4957-96b9-efa47269c720","Type":"ContainerDied","Data":"f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5"}
Sep 09 13:19:10.918469 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.918422    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x" event={"ID":"af4a1b42-c978-4957-96b9-efa47269c720","Type":"ContainerDied","Data":"0560b3d83e9b036494b14b88fd5017b9d33e3f2d26bc694df7d4383fa3536ab7"}
Sep 09 13:19:10.918469 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.918451    2580 scope.go:117] "RemoveContainer" containerID="f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5"
Sep 09 13:19:10.919790 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.919761    2580 generic.go:356] "Generic (PLEG): container finished" podID="388cde2e-18cb-44cb-b4c4-7e3726160f01" containerID="2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986" exitCode=0
Sep 09 13:19:10.919920 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.919802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x" event={"ID":"388cde2e-18cb-44cb-b4c4-7e3726160f01","Type":"ContainerDied","Data":"2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986"}
Sep 09 13:19:10.919920 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.919812    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"
Sep 09 13:19:10.919920 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.919880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x" event={"ID":"388cde2e-18cb-44cb-b4c4-7e3726160f01","Type":"ContainerDied","Data":"3c54afc12a40e091474f34246bf4c26cccfc37ab6be7de4e718ffa527d6bf02d"}
Sep 09 13:19:10.929076 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.929033    2580 scope.go:117] "RemoveContainer" containerID="f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5"
Sep 09 13:19:10.929980 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:19:10.929953    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5\": container with ID starting with f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5 not found: ID does not exist" containerID="f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5"
Sep 09 13:19:10.930101 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.929988    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5"} err="failed to get container status \"f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5\": rpc error: code = NotFound desc = could not find container \"f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5\": container with ID starting with f2b47988dbc6c5f82ca49d555ccde4ddb6261412f0de47247d06969668998ce5 not found: ID does not exist"
Sep 09 13:19:10.930101 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.930008    2580 scope.go:117] "RemoveContainer" containerID="2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986"
Sep 09 13:19:10.938606 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.938592    2580 scope.go:117] "RemoveContainer" containerID="2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986"
Sep 09 13:19:10.938876 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:19:10.938854    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986\": container with ID starting with 2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986 not found: ID does not exist" containerID="2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986"
Sep 09 13:19:10.938948 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.938886    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986"} err="failed to get container status \"2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986\": rpc error: code = NotFound desc = could not find container \"2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986\": container with ID starting with 2439d83efceb28edc0bb74c471e1e82f11b85231008f090b7cc3b696510a4986 not found: ID does not exist"
Sep 09 13:19:10.944756 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.944735    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"]
Sep 09 13:19:10.948524 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.948502    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-650a99-5fc8648888-z9h2x"]
Sep 09 13:19:10.958034 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.958017    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"]
Sep 09 13:19:10.961642 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:10.961624    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-650a99-79c9f67457-jd67x"]
Sep 09 13:19:11.591852 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:11.591798    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="388cde2e-18cb-44cb-b4c4-7e3726160f01" path="/var/lib/kubelet/pods/388cde2e-18cb-44cb-b4c4-7e3726160f01/volumes"
Sep 09 13:19:11.592202 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:11.592156    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af4a1b42-c978-4957-96b9-efa47269c720" path="/var/lib/kubelet/pods/af4a1b42-c978-4957-96b9-efa47269c720/volumes"
Sep 09 13:19:13.072897 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:13.072863    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:19:21.066078 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:19:21.066031    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:20:43.774258 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:20:43.774227    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:21:07.872343 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:21:07.872304    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:21:15.770921 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:21:15.770888    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:21:24.186755 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:21:24.186718    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:21:37.930043 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:21:37.929974    2580 scope.go:117] "RemoveContainer" containerID="44d0ed77fb19a9d7b5fd612c8e078e6ad00904e62a6b6919fb84faeb3a0cf050"
Sep 09 13:21:42.380399 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:21:42.380362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:21:48.983365 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:21:48.983326    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:23:20.670717 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:20.670683    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:23:26.770451 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:26.770403    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:23:30.974322 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.974281    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"]
Sep 09 13:23:30.974913 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.974894    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="614fef1f-23a0-48c6-8b78-297cf37d1611" containerName="payload-processing"
Sep 09 13:23:30.974974 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.974917    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="614fef1f-23a0-48c6-8b78-297cf37d1611" containerName="payload-processing"
Sep 09 13:23:30.974974 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.974930    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4a1b42-c978-4957-96b9-efa47269c720" containerName="payload-processing"
Sep 09 13:23:30.974974 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.974939    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4a1b42-c978-4957-96b9-efa47269c720" containerName="payload-processing"
Sep 09 13:23:30.974974 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.974949    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4ff669-ffb9-4143-84ad-b04b95e31a68" containerName="maas-api"
Sep 09 13:23:30.974974 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.974957    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4ff669-ffb9-4143-84ad-b04b95e31a68" containerName="maas-api"
Sep 09 13:23:30.974974 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.974972    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="388cde2e-18cb-44cb-b4c4-7e3726160f01" containerName="maas-api"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.974980    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="388cde2e-18cb-44cb-b4c4-7e3726160f01" containerName="maas-api"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.974993    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98fa770d-4e44-4f31-88a1-ceed95d99d16" containerName="maas-api"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975001    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98fa770d-4e44-4f31-88a1-ceed95d99d16" containerName="maas-api"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975039    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0a2021-5883-44d5-b45f-a2d5af66f873" containerName="payload-processing"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975049    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0a2021-5883-44d5-b45f-a2d5af66f873" containerName="payload-processing"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975063    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8dddb06-c245-407e-996d-8f7021f48375" containerName="payload-processing"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975071    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8dddb06-c245-407e-996d-8f7021f48375" containerName="payload-processing"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975160    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="614fef1f-23a0-48c6-8b78-297cf37d1611" containerName="payload-processing"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975172    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0a2021-5883-44d5-b45f-a2d5af66f873" containerName="payload-processing"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975180    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98fa770d-4e44-4f31-88a1-ceed95d99d16" containerName="maas-api"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975191    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4ff669-ffb9-4143-84ad-b04b95e31a68" containerName="maas-api"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975198    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4a1b42-c978-4957-96b9-efa47269c720" containerName="payload-processing"
Sep 09 13:23:30.975204 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975207    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="388cde2e-18cb-44cb-b4c4-7e3726160f01" containerName="maas-api"
Sep 09 13:23:30.975589 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.975218    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8dddb06-c245-407e-996d-8f7021f48375" containerName="payload-processing"
Sep 09 13:23:30.978896 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.978877    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:30.981404 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.981384    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-947aaaa2-kserve-self-signed-certs\""
Sep 09 13:23:30.987613 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:30.987587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"]
Sep 09 13:23:31.017235 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.017205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-home\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.017415 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.017239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-model-cache\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.017415 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.017272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-dshm\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.017415 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.017391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tmp-dir\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.017561 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.017456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tls-certs\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.017561 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.017506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47tpp\" (UniqueName: \"kubernetes.io/projected/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-kube-api-access-47tpp\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.119046 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.119011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-dshm\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.119236 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.119073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tmp-dir\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.119236 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.119105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tls-certs\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.119236 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.119141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47tpp\" (UniqueName: \"kubernetes.io/projected/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-kube-api-access-47tpp\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.119406 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.119270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-home\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.119406 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.119324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-model-cache\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.119541 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.119520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tmp-dir\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.119591 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.119563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-home\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.119692 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.119664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-model-cache\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.121523 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.121504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-dshm\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.121682 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.121667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tls-certs\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.126993 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.126972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47tpp\" (UniqueName: \"kubernetes.io/projected/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-kube-api-access-47tpp\") pod \"e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") " pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.288311 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.288248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:31.557614 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:31.557588    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"]
Sep 09 13:23:31.559978 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:23:31.559947    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod485e73c9_c8a7_4698_ae1a_65aa29ebd72c.slice/crio-af40c8e74c3a6b055d3e4cb42a95358a67b45d09d2c4f5773c32c08c57ba6d78 WatchSource:0}: Error finding container af40c8e74c3a6b055d3e4cb42a95358a67b45d09d2c4f5773c32c08c57ba6d78: Status 404 returned error can't find the container with id af40c8e74c3a6b055d3e4cb42a95358a67b45d09d2c4f5773c32c08c57ba6d78
Sep 09 13:23:32.169500 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:32.169468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n" event={"ID":"485e73c9-c8a7-4698-ae1a-65aa29ebd72c","Type":"ContainerStarted","Data":"32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918"}
Sep 09 13:23:32.169500 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:32.169505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n" event={"ID":"485e73c9-c8a7-4698-ae1a-65aa29ebd72c","Type":"ContainerStarted","Data":"af40c8e74c3a6b055d3e4cb42a95358a67b45d09d2c4f5773c32c08c57ba6d78"}
Sep 09 13:23:32.185795 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:32.185751    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n" podStartSLOduration=2.185737049 podStartE2EDuration="2.185737049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:23:32.184196898 +0000 UTC m=+2095.104088239" watchObservedRunningTime="2026-09-09 13:23:32.185737049 +0000 UTC m=+2095.105628387"
Sep 09 13:23:32.979686 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:32.979649    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:23:38.253174 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:38.253135    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 13:23:38.259950 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:38.259924    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cq52_8dbba43e-056a-4448-85cd-36f295e6f710/ovn-acl-logging/0.log"
Sep 09 13:23:41.289416 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:41.289377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:41.289797 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:41.289428    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:41.302480 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:41.302454    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:42.227432 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:42.227399    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:23:44.277597 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:44.277565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:23:57.393618 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:57.393583    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"]
Sep 09 13:23:59.467620 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:23:59.467581    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:24:08.276710 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:08.276675    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:24:12.413241 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.413176    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n" podUID="485e73c9-c8a7-4698-ae1a-65aa29ebd72c" containerName="main" containerID="cri-o://32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918" gracePeriod=45
Sep 09 13:24:12.660507 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.660483    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:24:12.687151 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687085    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-model-cache\") pod \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") "
Sep 09 13:24:12.687285 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687174    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-dshm\") pod \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") "
Sep 09 13:24:12.687285 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687220    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tls-certs\") pod \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") "
Sep 09 13:24:12.687285 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687248    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tmp-dir\") pod \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") "
Sep 09 13:24:12.687285 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47tpp\" (UniqueName: \"kubernetes.io/projected/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-kube-api-access-47tpp\") pod \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") "
Sep 09 13:24:12.687518 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-home\") pod \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\" (UID: \"485e73c9-c8a7-4698-ae1a-65aa29ebd72c\") "
Sep 09 13:24:12.687518 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687363    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-model-cache" (OuterVolumeSpecName: "model-cache") pod "485e73c9-c8a7-4698-ae1a-65aa29ebd72c" (UID: "485e73c9-c8a7-4698-ae1a-65aa29ebd72c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:24:12.687637 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "485e73c9-c8a7-4698-ae1a-65aa29ebd72c" (UID: "485e73c9-c8a7-4698-ae1a-65aa29ebd72c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:24:12.687695 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687646    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-home" (OuterVolumeSpecName: "home") pod "485e73c9-c8a7-4698-ae1a-65aa29ebd72c" (UID: "485e73c9-c8a7-4698-ae1a-65aa29ebd72c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:24:12.687791 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687774    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-model-cache\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:24:12.687876 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687799    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tmp-dir\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:24:12.687876 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.687812    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-home\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:24:12.689487 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.689455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-kube-api-access-47tpp" (OuterVolumeSpecName: "kube-api-access-47tpp") pod "485e73c9-c8a7-4698-ae1a-65aa29ebd72c" (UID: "485e73c9-c8a7-4698-ae1a-65aa29ebd72c"). InnerVolumeSpecName "kube-api-access-47tpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:24:12.689986 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.689957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-dshm" (OuterVolumeSpecName: "dshm") pod "485e73c9-c8a7-4698-ae1a-65aa29ebd72c" (UID: "485e73c9-c8a7-4698-ae1a-65aa29ebd72c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:24:12.690087 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.689958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "485e73c9-c8a7-4698-ae1a-65aa29ebd72c" (UID: "485e73c9-c8a7-4698-ae1a-65aa29ebd72c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:24:12.788789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.788753    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-dshm\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:24:12.788789 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.788784    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-tls-certs\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:24:12.788997 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:12.788799    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47tpp\" (UniqueName: \"kubernetes.io/projected/485e73c9-c8a7-4698-ae1a-65aa29ebd72c-kube-api-access-47tpp\") on node \"ip-10-0-140-133.ec2.internal\" DevicePath \"\""
Sep 09 13:24:13.364157 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:13.364120    2580 generic.go:356] "Generic (PLEG): container finished" podID="485e73c9-c8a7-4698-ae1a-65aa29ebd72c" containerID="32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918" exitCode=0
Sep 09 13:24:13.364324 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:13.364168    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"
Sep 09 13:24:13.364324 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:13.364188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n" event={"ID":"485e73c9-c8a7-4698-ae1a-65aa29ebd72c","Type":"ContainerDied","Data":"32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918"}
Sep 09 13:24:13.364324 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:13.364228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n" event={"ID":"485e73c9-c8a7-4698-ae1a-65aa29ebd72c","Type":"ContainerDied","Data":"af40c8e74c3a6b055d3e4cb42a95358a67b45d09d2c4f5773c32c08c57ba6d78"}
Sep 09 13:24:13.364324 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:13.364249    2580 scope.go:117] "RemoveContainer" containerID="32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918"
Sep 09 13:24:13.373866 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:13.373850    2580 scope.go:117] "RemoveContainer" containerID="32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918"
Sep 09 13:24:13.374091 ip-10-0-140-133 kubenswrapper[2580]: E0909 13:24:13.374073    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918\": container with ID starting with 32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918 not found: ID does not exist" containerID="32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918"
Sep 09 13:24:13.374149 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:13.374099    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918"} err="failed to get container status \"32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918\": rpc error: code = NotFound desc = could not find container \"32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918\": container with ID starting with 32b454fd3351118f964c76da23845b355dcb4d9c2906e480df75f4797101f918 not found: ID does not exist"
Sep 09 13:24:13.385866 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:13.385844    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"]
Sep 09 13:24:13.390491 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:13.390472    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-947aaaa2-kserve-79bd8f789f-89v5n"]
Sep 09 13:24:13.592142 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:13.592112    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="485e73c9-c8a7-4698-ae1a-65aa29ebd72c" path="/var/lib/kubelet/pods/485e73c9-c8a7-4698-ae1a-65aa29ebd72c/volumes"
Sep 09 13:24:15.072360 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:15.072330    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:24:22.779456 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.779424    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:24:22.804765 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.804738    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"]
Sep 09 13:24:22.805151 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.805138    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="485e73c9-c8a7-4698-ae1a-65aa29ebd72c" containerName="main"
Sep 09 13:24:22.805203 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.805153    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="485e73c9-c8a7-4698-ae1a-65aa29ebd72c" containerName="main"
Sep 09 13:24:22.805239 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.805230    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="485e73c9-c8a7-4698-ae1a-65aa29ebd72c" containerName="main"
Sep 09 13:24:22.807276 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.807261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.810920 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.810896    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:24:22.811050 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.810903    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 13:24:22.811050 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.811037    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dbjk4\""
Sep 09 13:24:22.811168 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.811111    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 13:24:22.811229 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.811179    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 13:24:22.816355 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.816323    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"]
Sep 09 13:24:22.877154 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.877126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d764845a-b19e-43d0-a444-1c3213659aac-openshift-service-ca\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.877348 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.877178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wr68\" (UniqueName: \"kubernetes.io/projected/d764845a-b19e-43d0-a444-1c3213659aac-kube-api-access-4wr68\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.877348 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.877283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d764845a-b19e-43d0-a444-1c3213659aac-metrics-certs\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.877348 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.877320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d764845a-b19e-43d0-a444-1c3213659aac-maas-api-tls\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.978587 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.978561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d764845a-b19e-43d0-a444-1c3213659aac-openshift-service-ca\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.978749 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.978596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wr68\" (UniqueName: \"kubernetes.io/projected/d764845a-b19e-43d0-a444-1c3213659aac-kube-api-access-4wr68\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.978749 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.978639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d764845a-b19e-43d0-a444-1c3213659aac-metrics-certs\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.978749 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.978671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d764845a-b19e-43d0-a444-1c3213659aac-maas-api-tls\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.979279 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.979257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d764845a-b19e-43d0-a444-1c3213659aac-openshift-service-ca\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.981168 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.981149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d764845a-b19e-43d0-a444-1c3213659aac-maas-api-tls\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.981266 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.981169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d764845a-b19e-43d0-a444-1c3213659aac-metrics-certs\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:22.987337 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:22.987317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wr68\" (UniqueName: \"kubernetes.io/projected/d764845a-b19e-43d0-a444-1c3213659aac-kube-api-access-4wr68\") pod \"maas-api-6b9c84785b-lhxzg\" (UID: \"d764845a-b19e-43d0-a444-1c3213659aac\") " pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:23.119881 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:23.119859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:23.390144 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:23.390110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"]
Sep 09 13:24:23.396111 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:24:23.395682    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd764845a_b19e_43d0_a444_1c3213659aac.slice/crio-6c821928f74da332c97f99e51eedb7af250510319fcad7b7216d8bab9fcf2489 WatchSource:0}: Error finding container 6c821928f74da332c97f99e51eedb7af250510319fcad7b7216d8bab9fcf2489: Status 404 returned error can't find the container with id 6c821928f74da332c97f99e51eedb7af250510319fcad7b7216d8bab9fcf2489
Sep 09 13:24:23.396640 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:23.396620    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 13:24:23.411126 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:23.411105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg" event={"ID":"d764845a-b19e-43d0-a444-1c3213659aac","Type":"ContainerStarted","Data":"6c821928f74da332c97f99e51eedb7af250510319fcad7b7216d8bab9fcf2489"}
Sep 09 13:24:24.418617 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:24.418577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg" event={"ID":"d764845a-b19e-43d0-a444-1c3213659aac","Type":"ContainerStarted","Data":"cf835f70ee37c8d0165e2c71156ea17145affcbbecc6227857dab7f889299b55"}
Sep 09 13:24:24.419101 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:24.418724    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:24.436306 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:24.436261    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg" podStartSLOduration=1.912945736 podStartE2EDuration="2.436248011s" totalImagesPullingTime="523.302275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:24:24.435053387 +0000 UTC m=+2147.354944729" watchObservedRunningTime="2026-09-09 13:24:24.436248011 +0000 UTC m=+2147.356139350"
Sep 09 13:24:29.569549 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:29.569471    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:24:30.430086 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:30.430055    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b9c84785b-lhxzg"
Sep 09 13:24:35.467747 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:35.467708    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:24:42.170529 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:42.170483    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wh5m6"]
Sep 09 13:24:51.862309 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:51.862275    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-647774d94-2wgx4_540d3089-efae-4516-a6e1-c70d67f08eec/copy-manifests/0.log"
Sep 09 13:24:51.870092 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:51.870068    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-647774d94-2wgx4_540d3089-efae-4516-a6e1-c70d67f08eec/manager/0.log"
Sep 09 13:24:52.493168 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:52.493140    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b9c84785b-lhxzg_d764845a-b19e-43d0-a444-1c3213659aac/maas-api/0.log"
Sep 09 13:24:52.730471 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:52.730444    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29815980-glh9h_ba9dc355-7e54-48f0-9e43-dceaa36ba16f/cleanup/0.log"
Sep 09 13:24:54.169980 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:54.169951    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-dwc5b_297fb6db-f5f9-4711-abcc-1640a374b64f/kuadrant-console-plugin/0.log"
Sep 09 13:24:54.425194 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:54.425107    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-wh5m6_60a102b1-adad-4daa-a957-c8a4eba449a1/limitador/0.log"
Sep 09 13:24:55.174304 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:55.174270    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-768599446d-qthzs_38c5c960-e0c7-4df3-9a77-c9516507f8ec/kube-auth-proxy/0.log"
Sep 09 13:24:56.285202 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:56.285165    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-npr5j_bb87c7cd-e260-4632-ab78-cea8902c6e7f/main/0.log"
Sep 09 13:24:56.526938 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:56.526907    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9tgk4_4f26948a-bc1f-4e36-8a1a-9eb745e1f06a/main/0.log"
Sep 09 13:24:56.762094 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:56.762064    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-mqsqp_78fbf9d4-ad4d-4f64-94df-da1fef0da984/main/0.log"
Sep 09 13:24:56.876767 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:24:56.876718    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-jrw9n_f3c58d0c-938f-4a0c-bb92-33a77cc8e677/main/0.log"
Sep 09 13:25:05.924959 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:05.924919    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7jcs4_2014baf8-52ee-420c-89d0-205989b32b87/global-pull-secret-syncer/0.log"
Sep 09 13:25:06.222425 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:06.222346    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qq7xl_28b841c2-6400-4ee4-bdf6-8f238cc60545/konnectivity-agent/0.log"
Sep 09 13:25:06.335340 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:06.335309    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-133.ec2.internal_5785166f811993e15a031323bc2330c5/haproxy/0.log"
Sep 09 13:25:10.666948 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:10.666919    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-dwc5b_297fb6db-f5f9-4711-abcc-1640a374b64f/kuadrant-console-plugin/0.log"
Sep 09 13:25:10.749438 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:10.749400    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-wh5m6_60a102b1-adad-4daa-a957-c8a4eba449a1/limitador/0.log"
Sep 09 13:25:12.400805 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:12.400773    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9322d04f-d442-45ac-8932-122cb25bcadd/alertmanager/0.log"
Sep 09 13:25:12.424726 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:12.424701    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9322d04f-d442-45ac-8932-122cb25bcadd/config-reloader/0.log"
Sep 09 13:25:12.447796 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:12.447767    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9322d04f-d442-45ac-8932-122cb25bcadd/kube-rbac-proxy-web/0.log"
Sep 09 13:25:12.475767 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:12.475743    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9322d04f-d442-45ac-8932-122cb25bcadd/kube-rbac-proxy/0.log"
Sep 09 13:25:12.503217 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:12.503192    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9322d04f-d442-45ac-8932-122cb25bcadd/kube-rbac-proxy-metric/0.log"
Sep 09 13:25:12.527873 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:12.527847    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9322d04f-d442-45ac-8932-122cb25bcadd/prom-label-proxy/0.log"
Sep 09 13:25:12.550997 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:12.550977    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9322d04f-d442-45ac-8932-122cb25bcadd/init-config-reloader/0.log"
Sep 09 13:25:13.063052 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.063021    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qb94p_53f9d380-aeec-4335-995d-5cd1df5e0026/node-exporter/0.log"
Sep 09 13:25:13.091951 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.091873    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qb94p_53f9d380-aeec-4335-995d-5cd1df5e0026/kube-rbac-proxy/0.log"
Sep 09 13:25:13.116838 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.116803    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qb94p_53f9d380-aeec-4335-995d-5cd1df5e0026/init-textfile/0.log"
Sep 09 13:25:13.146783 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.146755    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-l5zws_8345b31d-f1bd-42ee-bf5e-b24dc175ce63/kube-rbac-proxy-main/0.log"
Sep 09 13:25:13.186094 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.186071    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-l5zws_8345b31d-f1bd-42ee-bf5e-b24dc175ce63/kube-rbac-proxy-self/0.log"
Sep 09 13:25:13.211239 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.211216    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-l5zws_8345b31d-f1bd-42ee-bf5e-b24dc175ce63/openshift-state-metrics/0.log"
Sep 09 13:25:13.251533 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.251508    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_58b06d01-fb3f-4ca8-85f9-ef2732628b1f/prometheus/0.log"
Sep 09 13:25:13.275262 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.275236    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_58b06d01-fb3f-4ca8-85f9-ef2732628b1f/config-reloader/0.log"
Sep 09 13:25:13.300053 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.300034    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_58b06d01-fb3f-4ca8-85f9-ef2732628b1f/thanos-sidecar/0.log"
Sep 09 13:25:13.325236 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.325217    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_58b06d01-fb3f-4ca8-85f9-ef2732628b1f/kube-rbac-proxy-web/0.log"
Sep 09 13:25:13.349669 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.349614    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_58b06d01-fb3f-4ca8-85f9-ef2732628b1f/kube-rbac-proxy/0.log"
Sep 09 13:25:13.379145 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.379119    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_58b06d01-fb3f-4ca8-85f9-ef2732628b1f/kube-rbac-proxy-thanos/0.log"
Sep 09 13:25:13.406605 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.406580    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_58b06d01-fb3f-4ca8-85f9-ef2732628b1f/init-config-reloader/0.log"
Sep 09 13:25:13.441749 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.441722    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-9r5z6_8086d1db-13a0-41e5-8775-2a1bd44137c2/prometheus-operator/0.log"
Sep 09 13:25:13.466095 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.466075    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-9r5z6_8086d1db-13a0-41e5-8775-2a1bd44137c2/kube-rbac-proxy/0.log"
Sep 09 13:25:13.503001 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.502977    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-7b7k2_f4d1687c-864d-45fa-b24f-ce8af11ed168/prometheus-operator-admission-webhook/0.log"
Sep 09 13:25:13.550375 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.550355    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58666c69f4-294bn_43408383-4017-450f-98ba-356093595129/telemeter-client/0.log"
Sep 09 13:25:13.575571 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.575549    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58666c69f4-294bn_43408383-4017-450f-98ba-356093595129/reload/0.log"
Sep 09 13:25:13.598131 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.598110    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58666c69f4-294bn_43408383-4017-450f-98ba-356093595129/kube-rbac-proxy/0.log"
Sep 09 13:25:13.631932 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.631867    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f4b64f9fb-gmdz6_a65c5be9-4322-4a18-be47-6a49e215ad19/thanos-query/0.log"
Sep 09 13:25:13.660591 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.660567    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f4b64f9fb-gmdz6_a65c5be9-4322-4a18-be47-6a49e215ad19/kube-rbac-proxy-web/0.log"
Sep 09 13:25:13.691282 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.691258    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f4b64f9fb-gmdz6_a65c5be9-4322-4a18-be47-6a49e215ad19/kube-rbac-proxy/0.log"
Sep 09 13:25:13.714610 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.714588    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f4b64f9fb-gmdz6_a65c5be9-4322-4a18-be47-6a49e215ad19/prom-label-proxy/0.log"
Sep 09 13:25:13.737112 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.737086    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f4b64f9fb-gmdz6_a65c5be9-4322-4a18-be47-6a49e215ad19/kube-rbac-proxy-rules/0.log"
Sep 09 13:25:13.760113 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:13.760086    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f4b64f9fb-gmdz6_a65c5be9-4322-4a18-be47-6a49e215ad19/kube-rbac-proxy-metrics/0.log"
Sep 09 13:25:15.607573 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.607539    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"]
Sep 09 13:25:15.611240 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.611220    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.613914 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.613888    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sv79w\"/\"kube-root-ca.crt\""
Sep 09 13:25:15.614018 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.613957    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sv79w\"/\"openshift-service-ca.crt\""
Sep 09 13:25:15.614987 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.614967    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sv79w\"/\"default-dockercfg-d6hxv\""
Sep 09 13:25:15.621336 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.621309    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"]
Sep 09 13:25:15.734917 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.734888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-proc\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.735090 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.734955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-podres\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.735090 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.735016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-lib-modules\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.735090 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.735063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-sys\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.735216 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.735160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbngc\" (UniqueName: \"kubernetes.io/projected/ca8f8175-79dd-4f89-966c-540355188483-kube-api-access-jbngc\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.835969 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.835933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-sys\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.836147 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.836031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbngc\" (UniqueName: \"kubernetes.io/projected/ca8f8175-79dd-4f89-966c-540355188483-kube-api-access-jbngc\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.836147 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.836062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-proc\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.836147 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.836070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-sys\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.836147 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.836115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-podres\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.836147 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.836135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-proc\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.836147 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.836145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-lib-modules\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.836357 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.836242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-podres\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.836357 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.836262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca8f8175-79dd-4f89-966c-540355188483-lib-modules\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.845133 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.845102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbngc\" (UniqueName: \"kubernetes.io/projected/ca8f8175-79dd-4f89-966c-540355188483-kube-api-access-jbngc\") pod \"perf-node-gather-daemonset-zh92g\" (UID: \"ca8f8175-79dd-4f89-966c-540355188483\") " pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:15.921341 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:15.921252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:16.193835 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:16.193733    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"]
Sep 09 13:25:16.197481 ip-10-0-140-133 kubenswrapper[2580]: W0909 13:25:16.197453    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podca8f8175_79dd_4f89_966c_540355188483.slice/crio-9f9da6aff122c9ae824144b4c3f450a49752f29d009eddb0f4f369b08ec43be1 WatchSource:0}: Error finding container 9f9da6aff122c9ae824144b4c3f450a49752f29d009eddb0f4f369b08ec43be1: Status 404 returned error can't find the container with id 9f9da6aff122c9ae824144b4c3f450a49752f29d009eddb0f4f369b08ec43be1
Sep 09 13:25:16.535319 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:16.535219    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-mjfdh_f09261ce-ec57-49c9-b05c-cdf587fc4853/download-server/0.log"
Sep 09 13:25:16.666165 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:16.666133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g" event={"ID":"ca8f8175-79dd-4f89-966c-540355188483","Type":"ContainerStarted","Data":"263d965fca292828f48918ea3ad091b61a774d2a732438d1e44c9f6b8cc854ee"}
Sep 09 13:25:16.666552 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:16.666188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g" event={"ID":"ca8f8175-79dd-4f89-966c-540355188483","Type":"ContainerStarted","Data":"9f9da6aff122c9ae824144b4c3f450a49752f29d009eddb0f4f369b08ec43be1"}
Sep 09 13:25:16.666552 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:16.666221    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:16.688422 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:16.688367    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g" podStartSLOduration=1.68835084 podStartE2EDuration="1.68835084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:25:16.687796872 +0000 UTC m=+2199.607688212" watchObservedRunningTime="2026-09-09 13:25:16.68835084 +0000 UTC m=+2199.608242179"
Sep 09 13:25:18.463305 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:18.463271    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vhfsf_c3a0dfbf-070e-4c73-afd5-be6a620d8bb2/dns/0.log"
Sep 09 13:25:18.490241 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:18.490210    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vhfsf_c3a0dfbf-070e-4c73-afd5-be6a620d8bb2/kube-rbac-proxy/0.log"
Sep 09 13:25:18.627210 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:18.627177    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qp9gk_ffd649fb-0687-45dd-9acc-4a8d232f144d/dns-node-resolver/0.log"
Sep 09 13:25:19.190627 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:19.190594    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b45d9f7f5-bzzcr_95b8a79e-17b1-4b49-9cb9-bfb90cef9672/registry/0.log"
Sep 09 13:25:19.269420 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:19.269386    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tgvsf_34c5ccd1-df9d-4b19-b2ae-93d5827baef1/node-ca/0.log"
Sep 09 13:25:20.682667 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:20.682640    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-768599446d-qthzs_38c5c960-e0c7-4df3-9a77-c9516507f8ec/kube-auth-proxy/0.log"
Sep 09 13:25:21.895945 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:21.895910    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p8d9s_7faebb79-50d6-4521-b829-ba8c91fb8a9b/serve-healthcheck-canary/0.log"
Sep 09 13:25:22.532029 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:22.532000    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gtq25_739e03ea-ddf4-4d97-9552-851e87b324da/kube-rbac-proxy/0.log"
Sep 09 13:25:22.556762 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:22.556739    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gtq25_739e03ea-ddf4-4d97-9552-851e87b324da/exporter/0.log"
Sep 09 13:25:22.583998 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:22.583973    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gtq25_739e03ea-ddf4-4d97-9552-851e87b324da/extractor/0.log"
Sep 09 13:25:22.681683 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:22.681654    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sv79w/perf-node-gather-daemonset-zh92g"
Sep 09 13:25:25.968606 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:25.968516    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-647774d94-2wgx4_540d3089-efae-4516-a6e1-c70d67f08eec/manager/0.log"
Sep 09 13:25:25.991276 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:25.991251    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-647774d94-2wgx4_540d3089-efae-4516-a6e1-c70d67f08eec/copy-manifests/0.log"
Sep 09 13:25:26.568345 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:26.568304    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-768599446d-qthzs_38c5c960-e0c7-4df3-9a77-c9516507f8ec/kube-auth-proxy/0.log"
Sep 09 13:25:27.687662 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:27.687551    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-798b96659f-zpsks_bc9f0746-4d40-443c-b859-615ffb404eb3/manager/0.log"
Sep 09 13:25:28.665361 ip-10-0-140-133 kubenswrapper[2580]: I0909 13:25:28.665311    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-768599446d-qthzs_38c5c960-e0c7-4df3-9a77-c9516507f8ec/kube-auth-proxy/0.log"