Sep 09 12:17:37.735456 ip-10-0-133-152 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 12:17:37.735473 ip-10-0-133-152 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 12:17:37.735483 ip-10-0-133-152 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 12:17:37.735981 ip-10-0-133-152 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 12:17:47.923491 ip-10-0-133-152 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 12:17:47.923506 ip-10-0-133-152 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1fae0e7f1b574b209c857cfca96529ab --
Sep 09 12:19:38.484983 ip-10-0-133-152 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 12:19:38.913854 ip-10-0-133-152 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 12:19:38.913854 ip-10-0-133-152 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 12:19:38.913854 ip-10-0-133-152 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 12:19:38.913854 ip-10-0-133-152 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 12:19:38.913854 ip-10-0-133-152 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 12:19:38.915991 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.915892    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 12:19:38.919146 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919129    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:19:38.919146 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919146    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919149    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919152    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919155    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919159    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919165    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919168    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919171    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919174    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919176    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919180    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919182    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919185    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919187    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919190    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919192    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919195    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919197    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919200    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919202    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:19:38.919226 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919205    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919208    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919210    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919213    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919215    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919218    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919220    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919223    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919225    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919228    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919231    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919234    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919238    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919242    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919245    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919248    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919251    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919255    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919258    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919261    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:19:38.919741 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919263    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919266    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919269    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919272    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919275    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919278    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919280    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919283    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919285    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919288    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919290    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919293    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919295    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919298    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919301    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919304    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919306    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919309    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919313    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:19:38.920228 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919317    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919320    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919322    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919325    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919327    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919330    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919332    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919335    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919337    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919340    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919343    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919345    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919348    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919350    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919352    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919356    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919359    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919361    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919363    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919366    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:19:38.920949 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919369    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919371    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919374    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919376    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919379    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919381    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919383    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919387    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919390    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919393    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919395    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919398    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919400    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919402    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919405    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919407    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919410    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919413    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.919415    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920102    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:19:38.921627 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920108    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920111    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920114    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920117    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920120    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920123    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920125    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920128    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920131    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920134    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920136    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920139    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920141    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920144    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920146    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920149    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920152    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920154    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920157    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:19:38.922128 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920160    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920163    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920166    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920168    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920171    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920173    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920176    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920179    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920181    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920184    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920186    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920189    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920191    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920194    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920196    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920199    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920201    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920205    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920209    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920213    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:19:38.922602 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920215    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920218    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920221    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920223    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920225    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920228    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920231    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920233    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920236    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920239    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920241    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920244    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920248    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920251    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920255    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920258    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920262    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920264    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920267    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920270    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:19:38.923119 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920273    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920275    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920277    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920280    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920283    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920285    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920288    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920290    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920293    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920295    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920298    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920301    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920303    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920306    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920308    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920311    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920313    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920315    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920318    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920321    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:19:38.923613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920326    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920329    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920333    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920335    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920339    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920342    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920344    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920347    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920350    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920352    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920355    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920357    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920360    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920362    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920365    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920367    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920370    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920373    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.920375    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922085    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:19:38.924849 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922096    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922100    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922104    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922107    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922110    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922112    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922115    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922117    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922120    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922122    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922125    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922128    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922131    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922133    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922136    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922139    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922142    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922144    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922148    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922150    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:19:38.925345 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922153    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922156    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922159    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922161    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922164    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922166    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922169    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922171    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922174    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922176    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922178    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922181    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922185    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922190    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922193    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922197    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922200    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922202    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922205    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:19:38.925853 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922208    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922210    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922213    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922217    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922220    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922224    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922227    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922230    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922233    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922236    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922241    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922246    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922250    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922253    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922256    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922259    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922262    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922264    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922267    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:19:38.926321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922269    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922272    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922275    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922278    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922280    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922282    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922286    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922289    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922291    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922294    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922296    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922299    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922302    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922304    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922307    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922309    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922312    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922314    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922317    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922319    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:19:38.926810 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922322    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922325    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922327    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922330    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922333    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922335    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922338    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922340    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922342    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922345    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922348    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922350    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922352    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922355    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922357    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922359    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922362    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922367    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922370    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.922372    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:19:38.927337 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922458    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922496    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922504    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922511    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922518    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922523    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922529    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922535    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922540    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922543    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922546    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922550    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922553    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922556    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922559    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922562    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922565    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922569    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922576    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922578    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922582    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922585    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922588    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922592    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 12:19:38.927853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922595    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922598    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922601    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922604    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922607    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922612    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922615    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922618    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922623    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922626    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922628    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922631    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922634    2578 flags.go:64] FLAG: --enable-server="true"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922637    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922642    2578 flags.go:64] FLAG: --event-burst="100"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922645    2578 flags.go:64] FLAG: --event-qps="50"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922647    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922650    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922653    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922656    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922659    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922662    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922665    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922668    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922670    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 12:19:38.928433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922690    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922696    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922699    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922702    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922705    2578 flags.go:64] FLAG: --feature-gates=""
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922713    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922716    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922719    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922722    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922725    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922728    2578 flags.go:64] FLAG: --help="false"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922731    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-152.ec2.internal"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922736    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922739    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922742    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922745    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922749    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922751    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922754    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922757    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922760    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922762    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922766    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922769    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 09 12:19:38.929087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922771    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922774    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922777    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922780    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922782    2578 flags.go:64] FLAG: --lock-file=""
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922785    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922788    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922791    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922796    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922800    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922803    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922806    2578 flags.go:64] FLAG: --logging-format="text"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922809    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922813    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922815    2578 flags.go:64] FLAG: --manifest-url=""
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922818    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922823    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922826    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922830    2578 flags.go:64] FLAG: --max-pods="110"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922833    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922837    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922840    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922843    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922845    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922848    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 12:19:38.929708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922851    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922858    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922861    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922864    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922867    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922870    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922875    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922878    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922881    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922884    2578 flags.go:64] FLAG: --port="10250"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922887    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922890    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0be9ee11ebb16cec1"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922893    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922896    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922899    2578 flags.go:64] FLAG: --register-node="true"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922901    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922905    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922909    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922913    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922916    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922919    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922922    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922925    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922928    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922931    2578 flags.go:64] FLAG: --runonce="false"
Sep 09 12:19:38.930331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922933    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922937    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922941    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922944    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922947    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922950    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922953    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922956    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922958    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922961    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922964    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922967    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922970    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922973    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922975    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922981    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922984    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922986    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922990    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922993    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922996    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.922998    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.923001    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.923004    2578 flags.go:64] FLAG: --v="2"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.923009    2578 flags.go:64] FLAG: --version="false"
Sep 09 12:19:38.931001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.923013    2578 flags.go:64] FLAG: --vmodule=""
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.923018    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.923021    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923136    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923139    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923142    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923145    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923148    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923151    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923154    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923157    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923160    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923162    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923165    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923168    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923170    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923174    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923176    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923179    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923182    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:19:38.931613 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923184    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923187    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923190    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923192    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923195    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923197    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923200    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923202    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923205    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923207    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923210    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923213    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923216    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923218    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923221    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923224    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923227    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923229    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923232    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:19:38.932159 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923234    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923237    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923239    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923242    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923245    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923247    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923250    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923252    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923256    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923260    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923263    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923265    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923268    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923272    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923276    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923279    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923282    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923285    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923287    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:19:38.932631 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923290    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923293    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923296    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923298    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923301    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923303    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923306    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923309    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923311    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923314    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923316    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923319    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923321    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923324    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923328    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923331    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923334    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923337    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923340    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923342    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:19:38.933132 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923345    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923347    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923350    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923352    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923355    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923358    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923360    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923363    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923365    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923368    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923371    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923373    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923376    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923378    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923380    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923383    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923386    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923388    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923391    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923394    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:19:38.933625 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923396    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923399    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923402    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.923404    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.923414    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.931332    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.931351    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.931372    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.931381    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931458    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931463    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931466    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931469    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931472    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931475    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:19:38.934181 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931478    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931481    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931484    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931488    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931492    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931496    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931499    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931501    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931504    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931507    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931509    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931512    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931515    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931517    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931520    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931522    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931525    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931528    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931530    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:19:38.934567 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931533    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931535    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931538    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931540    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931545    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931547    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931550    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931552    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931555    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931557    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931560    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931563    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931566    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931568    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931571    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931573    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931576    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931578    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931581    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931583    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:19:38.935055 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931586    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931588    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931591    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931595    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931598    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931602    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931606    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931608    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931611    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931614    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931616    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931619    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931622    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931624    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931627    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931629    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931632    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931636    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931639    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:19:38.935535 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931641    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931644    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931646    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931649    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931651    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931654    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931656    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931659    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931661    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931664    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931667    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931669    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931672    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931692    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931695    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931698    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931702    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931706    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931709    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:19:38.936026 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931711    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931714    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931717    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931720    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931722    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931725    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931728    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931731    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931733    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931736    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931739    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931742    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931745    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931748    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931750    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:19:38.936485 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931753    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.931758    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931893    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931897    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931901    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931904    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931907    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931909    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931912    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931914    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931917    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931919    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931922    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931924    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931927    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:19:38.936867 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931929    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931931    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931934    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931936    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931939    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931942    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931944    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931947    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931949    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931952    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931955    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931957    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931960    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931963    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931966    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931969    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931971    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931974    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931976    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931978    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:19:38.937249 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931981    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931983    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931986    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931988    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931991    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931993    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931996    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.931998    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932000    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932003    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932005    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932008    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932010    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932013    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932015    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932018    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932020    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932022    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932025    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932027    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:19:38.937769 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932030    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932032    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932035    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932038    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932040    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932042    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932045    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932047    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932051    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932053    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932055    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932059    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932063    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932066    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932068    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932071    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932074    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932078    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932080    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:19:38.938264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932083    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932085    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932088    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932091    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932093    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932095    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932098    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932100    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932103    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932105    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932108    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932110    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932113    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932116    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932119    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932122    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932124    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932127    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932129    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932132    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:19:38.938825 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932134    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932137    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932140    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932142    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932145    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932147    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:38.932150    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.932156    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.932305    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.935602    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.936737    2578 server.go:1019] "Starting client certificate rotation"
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.936864    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 12:19:38.939320 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.936908    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 12:19:38.939819 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.939806    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 12:19:38.946361 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.946344    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 12:19:38.955381 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.955361    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 09 12:19:38.958598 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.958575    2578 log.go:25] "Validated CRI v1 image API"
Sep 09 12:19:38.959371 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.959354    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 12:19:38.961937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.961917    2578 fs.go:135] Filesystem UUIDs: map[0917f76c-d805-459b-b4c0-63a80e167d97:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a2c24f9c-76a6-493f-8c5c-318247cc9765:/dev/nvme0n1p3]
Sep 09 12:19:38.961984 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.961938    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:30 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 12:19:38.968569 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.968421    2578 manager.go:216] Machine: {Timestamp:2026-09-09 12:19:38.966915853 +0000 UTC m=+0.377051150 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2999206 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26a4f73ad4e8277a1fa2ae50a2dd19 SystemUUID:ec26a4f7-3ad4-e827-7a1f-a2ae50a2dd19 BootID:1fae0e7f-1b57-4b20-9c85-7cfca96529ab Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:30 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:50:f0:bd:0c:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:50:f0:bd:0c:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:b9:be:f0:c0:f2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 12:19:38.968656 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.968571    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 12:19:38.968705 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.968691    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 12:19:38.970473 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.970455    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 12:19:38.971569 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.971542    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 12:19:38.971743 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.971573    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-152.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 12:19:38.972404 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.972394    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 12:19:38.972434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.972408    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 12:19:38.972434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.972415    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 12:19:38.972434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.972430    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 12:19:38.973703 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.973694    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 12:19:38.977386 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.977375    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 12:19:38.977514 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.977505    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 12:19:38.977543 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.977537    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 12:19:38.977570 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.977550    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 12:19:38.977597 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.977577    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 12:19:38.977627 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.977601    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 09 12:19:38.977627 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.977613    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 12:19:38.978111 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.978098    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 12:19:38.978188 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.978119    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 12:19:38.980546 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.980513    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 12:19:38.983873 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.983859    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 12:19:38.983918 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.983890    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 12:19:38.985177 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985162    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 12:19:38.985261 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985183    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 12:19:38.985261 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985193    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 12:19:38.985261 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985201    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 12:19:38.985261 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985212    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 12:19:38.985261 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985219    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 12:19:38.985261 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985237    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 12:19:38.985261 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985245    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 12:19:38.985442 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985264    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 12:19:38.985442 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985273    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 12:19:38.985442 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985284    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 12:19:38.985442 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.985289    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 12:19:38.986981 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.986968    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 12:19:38.987055 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.986983    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 12:19:38.990364 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.990343    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-152.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 12:19:38.990433 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:38.990383    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-152.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 12:19:38.991531 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:38.991513    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 12:19:38.992555 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.992541    2578 server.go:1268] "Started kubelet"
Sep 09 12:19:38.992754 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.992722    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 12:19:38.993414 ip-10-0-133-152 systemd[1]: Started Kubernetes Kubelet.
Sep 09 12:19:38.993872 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.993521    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 12:19:38.993872 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.993606    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 12:19:38.995081 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.995065    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 12:19:38.995450 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:38.995436    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 12:19:38.997086 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:38.997068    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 12:19:39.001098 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.001084    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 12:19:39.001889 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.001874    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 12:19:39.002017 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.001997    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:39.002547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.002526    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 12:19:39.002547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.002546    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 12:19:39.002688 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.002531    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 12:19:39.002740 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.002732    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 12:19:39.002783 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.002741    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 12:19:39.002883 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.002867    2578 factory.go:55] Registering systemd factory
Sep 09 12:19:39.003050 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.002889    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 09 12:19:39.003143 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.003133    2578 factory.go:153] Registering CRI-O factory
Sep 09 12:19:39.003189 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.003146    2578 factory.go:223] Registration of the crio container factory successfully
Sep 09 12:19:39.003234 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.003197    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 12:19:39.003234 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.003217    2578 factory.go:103] Registering Raw factory
Sep 09 12:19:39.003234 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.003230    2578 manager.go:1196] Started watching for new ooms in manager
Sep 09 12:19:39.003809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.003793    2578 manager.go:349] Starting recovery of all containers
Sep 09 12:19:39.008825 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.008770    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-152.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 12:19:39.009026 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.008999    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 12:19:39.010320 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.009083    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-152.ec2.internal.18d3a65930814002  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-152.ec2.internal,UID:ip-10-0-133-152.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-152.ec2.internal,},FirstTimestamp:2026-09-09 12:19:38.992295938 +0000 UTC m=+0.402431234,LastTimestamp:2026-09-09 12:19:38.992295938 +0000 UTC m=+0.402431234,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-152.ec2.internal,}"
Sep 09 12:19:39.011705 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.011655    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 12:19:39.016614 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.016596    2578 manager.go:354] Recovery completed
Sep 09 12:19:39.019471 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.019448    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t5f4z"
Sep 09 12:19:39.022232 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.022219    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:19:39.023977 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.023958    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:19:39.024046 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.023996    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:19:39.024046 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.024007    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:19:39.024525 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.024510    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 12:19:39.024525 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.024523    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 12:19:39.024629 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.024537    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 12:19:39.024629 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.024524    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:19:39.025235 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.025222    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:19:39.025284 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.025245    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:19:39.025284 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.025254    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:19:39.025638 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.025626    2578 policy_none.go:49] "None policy: Start"
Sep 09 12:19:39.025698 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.025643    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 12:19:39.025698 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.025653    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 12:19:39.026033 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.025973    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-152.ec2.internal.18d3a6593264b51c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-152.ec2.internal,UID:ip-10-0-133-152.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-152.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-152.ec2.internal,},FirstTimestamp:2026-09-09 12:19:39.023979804 +0000 UTC m=+0.434115108,LastTimestamp:2026-09-09 12:19:39.023979804 +0000 UTC m=+0.434115108,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-152.ec2.internal,}"
Sep 09 12:19:39.026449 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.026439    2578 policy_none.go:47] "Start"
Sep 09 12:19:39.027548 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.027536    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t5f4z"
Sep 09 12:19:39.064327 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.064304    2578 manager.go:337] "Starting Device Plugin manager"
Sep 09 12:19:39.064476 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.064353    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 12:19:39.064476 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.064366    2578 server.go:87] "Starting device plugin registration server"
Sep 09 12:19:39.064849 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.064723    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 12:19:39.064849 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.064736    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 12:19:39.065400 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.064897    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 12:19:39.065400 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.064991    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 12:19:39.065400 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.065000    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 12:19:39.065525 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.065435    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 12:19:39.065525 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.065505    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:39.112722 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.112670    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 12:19:39.112848 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.112741    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 12:19:39.112848 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.112764    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 12:19:39.112848 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.112805    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 12:19:39.114835 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.114814    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:19:39.164897 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.164816    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:19:39.165795 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.165779    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:19:39.165900 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.165814    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:19:39.165900 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.165828    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:19:39.165900 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.165861    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.174289 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.174273    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.174340 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.174296    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-152.ec2.internal\": node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:39.205402 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.205376    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:39.213585 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.213567    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-152.ec2.internal"]
Sep 09 12:19:39.213631 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.213626    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:19:39.215073 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.215053    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:19:39.215140 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.215085    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:19:39.215140 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.215095    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:19:39.216331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.216319    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:19:39.216453 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.216439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.216492 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.216468    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:19:39.217041 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.217024    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:19:39.217111 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.217047    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:19:39.217111 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.217032    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:19:39.217111 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.217082    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:19:39.217111 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.217056    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:19:39.217248 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.217099    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:19:39.218026 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.218012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.218079 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.218035    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:19:39.218668 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.218652    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:19:39.218742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.218698    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:19:39.218742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.218714    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:19:39.243342 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.243319    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-152.ec2.internal\" not found" node="ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.247641 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.247624    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-152.ec2.internal\" not found" node="ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.304483 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.304450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aea0489aa54ab9ac328abed7fe01c8c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal\" (UID: \"aea0489aa54ab9ac328abed7fe01c8c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.304483 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.304482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea0489aa54ab9ac328abed7fe01c8c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal\" (UID: \"aea0489aa54ab9ac328abed7fe01c8c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.304697 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.304506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3308eb29717fa7fe5c5ea5d97019ea1-config\") pod \"kube-apiserver-proxy-ip-10-0-133-152.ec2.internal\" (UID: \"a3308eb29717fa7fe5c5ea5d97019ea1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.305497 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.305473    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:39.405644 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.405594    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:39.405854 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.405657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aea0489aa54ab9ac328abed7fe01c8c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal\" (UID: \"aea0489aa54ab9ac328abed7fe01c8c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.405854 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.405718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea0489aa54ab9ac328abed7fe01c8c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal\" (UID: \"aea0489aa54ab9ac328abed7fe01c8c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.405854 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.405741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3308eb29717fa7fe5c5ea5d97019ea1-config\") pod \"kube-apiserver-proxy-ip-10-0-133-152.ec2.internal\" (UID: \"a3308eb29717fa7fe5c5ea5d97019ea1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.405854 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.405754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aea0489aa54ab9ac328abed7fe01c8c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal\" (UID: \"aea0489aa54ab9ac328abed7fe01c8c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.405854 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.405796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3308eb29717fa7fe5c5ea5d97019ea1-config\") pod \"kube-apiserver-proxy-ip-10-0-133-152.ec2.internal\" (UID: \"a3308eb29717fa7fe5c5ea5d97019ea1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.405854 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.405824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea0489aa54ab9ac328abed7fe01c8c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal\" (UID: \"aea0489aa54ab9ac328abed7fe01c8c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.506379 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.506303    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:39.545480 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.545453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.550502 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.550306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:39.606408 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.606376    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:39.706800 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.706771    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:39.807195 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.807131    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:39.907709 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:39.907664    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:39.936849 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.936822    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 12:19:39.937276 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:39.936977    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 12:19:40.008325 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:40.008298    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:40.031826 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:40.031786    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 12:14:40 +0000 UTC" deadline="2028-04-19 04:16:30.151281113 +0000 UTC"
Sep 09 12:19:40.031826 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:40.031820    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14103h56m50.119465872s"
Sep 09 12:19:40.061659 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:40.061613    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:19:40.108735 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:40.108701    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:40.166771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:40.166746    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:19:40.182725 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:40.182695    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea0489aa54ab9ac328abed7fe01c8c4.slice/crio-70b9f73d3ba39fcb510e2d86a4d7cd9c1fe9e544ef9d255b09aaa4f1ccdff3f3 WatchSource:0}: Error finding container 70b9f73d3ba39fcb510e2d86a4d7cd9c1fe9e544ef9d255b09aaa4f1ccdff3f3: Status 404 returned error can't find the container with id 70b9f73d3ba39fcb510e2d86a4d7cd9c1fe9e544ef9d255b09aaa4f1ccdff3f3
Sep 09 12:19:40.188441 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:40.188426    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:19:40.209640 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:40.209611    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:40.309988 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:40.309952    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:40.410370 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:40.410335    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-152.ec2.internal\" not found"
Sep 09 12:19:40.417582 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:40.417567    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:19:40.502572 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:40.502530    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:40.516467 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:40.516447    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 12:19:40.517697 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:40.517671    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-152.ec2.internal"
Sep 09 12:19:40.524163 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:40.524147    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 12:19:42.024833 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.024755    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:19:42.045460 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.045429    2578 apiserver.go:52] "Watching apiserver"
Sep 09 12:19:42.052415 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.052392    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 12:19:42.052793 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.052773    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-thh9h","openshift-network-operator/iptables-alerter-xh264","openshift-ovn-kubernetes/ovnkube-node-4nvnq","kube-system/kube-apiserver-proxy-ip-10-0-133-152.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2","openshift-cluster-node-tuning-operator/tuned-cwjkj","openshift-dns/node-resolver-zhgxx","openshift-image-registry/node-ca-t7kpc","openshift-multus/multus-additional-cni-plugins-nvm9j","openshift-network-diagnostics/network-check-target-kxk9k","kube-system/global-pull-secret-syncer-qdgrt","kube-system/konnectivity-agent-jf5dm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal","openshift-multus/multus-kc57k"]
Sep 09 12:19:42.055450 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.055422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:42.055543 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.055509    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:19:42.057726 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.057707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:42.057827 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.057770    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:19:42.060314 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.060287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.063177 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.062843    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 12:19:42.063177 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.062882    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 12:19:42.063177 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.062928    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ffcgm\""
Sep 09 12:19:42.063177 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.062992    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 12:19:42.063177 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.063069    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 12:19:42.063177 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.062928    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 12:19:42.063464 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.063188    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 12:19:42.063464 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.063286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.064763 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.064737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.065359 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.065328    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 12:19:42.065460 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.065388    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 12:19:42.065460 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.065429    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 12:19:42.065615 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.065601    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 12:19:42.065814 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.065800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-flrvr\""
Sep 09 12:19:42.066460 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.066440    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 12:19:42.066557 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.066499    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 12:19:42.067255 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.067080    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.067255 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.067108    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d9plq\""
Sep 09 12:19:42.067255 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.067130    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 12:19:42.068232 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.068071    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 12:19:42.069319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.069279    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f4f55\""
Sep 09 12:19:42.069319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.069304    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:19:42.069536 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.069522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.070369 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.070350    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 12:19:42.071252 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.071227    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 12:19:42.071361 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.071339    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 12:19:42.071492 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.071476    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bdhdk\""
Sep 09 12:19:42.072158 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.071829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t7kpc"
Sep 09 12:19:42.076682 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.074018    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 12:19:42.076682 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.074080    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 12:19:42.076682 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.074389    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 12:19:42.076682 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.074702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xh264"
Sep 09 12:19:42.076682 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.075320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q2v6p\""
Sep 09 12:19:42.076987 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.076938    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 12:19:42.077269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.077237    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:19:42.078136 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.078116    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cxdms\""
Sep 09 12:19:42.078590 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.078576    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 12:19:42.078798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.078779    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:42.078859 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.078844    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:19:42.079571 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-kubernetes\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.079626 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-var-lib-kubelet\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.079626 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvc5\" (UniqueName: \"kubernetes.io/projected/35627b06-c338-4ef3-85b9-dd480994f542-kube-api-access-tzvc5\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.079718 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-sys\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.079718 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19401fc8-ae1c-4249-9a75-d7aab8a904de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.079816 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:42.079816 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-run-ovn\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.079816 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-node-log\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.079816 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-ovnkube-config\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.080009 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079817    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 12:19:42.080009 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-run-k8s-cni-cncf-io\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.080009 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-device-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.080009 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-lib-modules\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.080009 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-tuned\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.080009 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-log-socket\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.080009 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.079999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-registration-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-systemd-units\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-cni-bin\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-var-lib-cni-multus\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-hostroot\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-socket-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19401fc8-ae1c-4249-9a75-d7aab8a904de-cni-binary-copy\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7kn\" (UniqueName: \"kubernetes.io/projected/19401fc8-ae1c-4249-9a75-d7aab8a904de-kube-api-access-rc7kn\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-ovn-node-metrics-cert\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07004918-6af8-49a9-afc6-48bfb1fe376c-tmp\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-etc-openvswitch\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.080347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-cni-dir\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-var-lib-cni-bin\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jlf4\" (UniqueName: \"kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4\") pod \"network-check-target-kxk9k\" (UID: \"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610\") " pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlcxw\" (UniqueName: \"kubernetes.io/projected/07004918-6af8-49a9-afc6-48bfb1fe376c-kube-api-access-xlcxw\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-var-lib-kubelet\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-host\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-sys-fs\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-sysctl-d\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-sysctl-conf\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-run-netns\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080759    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zhgxx"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-run-multus-certs\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19401fc8-ae1c-4249-9a75-d7aab8a904de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-kubelet\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5p66\" (UniqueName: \"kubernetes.io/projected/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-kube-api-access-j5p66\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.080933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-modprobe-d\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-cni-netd\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-socket-dir-parent\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19401fc8-ae1c-4249-9a75-d7aab8a904de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d09d81f3-b038-48e2-88c2-6ab36ddefce2-dbus\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.080999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-cnibin\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-cni-binary-copy\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-conf-dir\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19401fc8-ae1c-4249-9a75-d7aab8a904de-system-cni-dir\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19401fc8-ae1c-4249-9a75-d7aab8a904de-cnibin\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-slash\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-run-netns\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-env-overrides\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d09d81f3-b038-48e2-88c2-6ab36ddefce2-kubelet-config\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-var-lib-openvswitch\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-run-openvswitch\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.081768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-ovnkube-script-lib\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.082452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwbm\" (UniqueName: \"kubernetes.io/projected/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-kube-api-access-rfwbm\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.082452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-systemd\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.082452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-run\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.082452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-run-systemd\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.082452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-system-cni-dir\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.082452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-daemon-config\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.082452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-sysconfig\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.082452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.081502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-etc-kubernetes\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.083659 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.082614    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tmdcd\""
Sep 09 12:19:42.083659 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.082627    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 12:19:42.083659 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.082730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:19:42.083659 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.082795    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 12:19:42.084817 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.084693    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t5bnl\""
Sep 09 12:19:42.084817 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.084744    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 12:19:42.084974 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.084839    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 12:19:42.102140 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.102120    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4dsvm"
Sep 09 12:19:42.112202 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.112180    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4dsvm"
Sep 09 12:19:42.146152 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:42.146129    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3308eb29717fa7fe5c5ea5d97019ea1.slice/crio-bc7959ee97a4b4dbddc358ce61c4b6e01e9b3db0322d8160de372a540f75a45b WatchSource:0}: Error finding container bc7959ee97a4b4dbddc358ce61c4b6e01e9b3db0322d8160de372a540f75a45b: Status 404 returned error can't find the container with id bc7959ee97a4b4dbddc358ce61c4b6e01e9b3db0322d8160de372a540f75a45b
Sep 09 12:19:42.170701 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.170674    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-sys\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19401fc8-ae1c-4249-9a75-d7aab8a904de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-run-ovn\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-node-log\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-ovnkube-config\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-run-k8s-cni-cncf-io\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-device-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-lib-modules\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-tuned\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-log-socket\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f087b26-7171-44d4-a216-7c223ebc6e4b-konnectivity-ca\") pod \"konnectivity-agent-jf5dm\" (UID: \"4f087b26-7171-44d4-a216-7c223ebc6e4b\") " pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-run-k8s-cni-cncf-io\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-node-log\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-registration-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.183765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-device-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-registration-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-systemd-units\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-sys\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-cni-bin\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-var-lib-cni-multus\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-var-lib-cni-multus\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-hostroot\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-systemd-units\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.182966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bb40332-7f47-480d-8fef-7be82e907b6a-host\") pod \"node-ca-t7kpc\" (UID: \"6bb40332-7f47-480d-8fef-7be82e907b6a\") " pod="openshift-image-registry/node-ca-t7kpc"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-socket-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19401fc8-ae1c-4249-9a75-d7aab8a904de-cni-binary-copy\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7kn\" (UniqueName: \"kubernetes.io/projected/19401fc8-ae1c-4249-9a75-d7aab8a904de-kube-api-access-rc7kn\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-socket-dir\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-lib-modules\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.184624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-ovnkube-config\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-ovn-node-metrics-cert\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-hostroot\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19401fc8-ae1c-4249-9a75-d7aab8a904de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmt57\" (UniqueName: \"kubernetes.io/projected/b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3-kube-api-access-lmt57\") pod \"node-resolver-zhgxx\" (UID: \"b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3\") " pod="openshift-dns/node-resolver-zhgxx"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-run-ovn\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.183325    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.183421    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret podName:d09d81f3-b038-48e2-88c2-6ab36ddefce2 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:42.683380241 +0000 UTC m=+3.026978173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret") pod "global-pull-secret-syncer-qdgrt" (UID: "d09d81f3-b038-48e2-88c2-6ab36ddefce2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-cni-bin\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4208c0ed-c841-4b55-9475-d8816e4c3ec8-iptables-alerter-script\") pod \"iptables-alerter-xh264\" (UID: \"4208c0ed-c841-4b55-9475-d8816e4c3ec8\") " pod="openshift-network-operator/iptables-alerter-xh264"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19401fc8-ae1c-4249-9a75-d7aab8a904de-cni-binary-copy\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-log-socket\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f087b26-7171-44d4-a216-7c223ebc6e4b-agent-certs\") pod \"konnectivity-agent-jf5dm\" (UID: \"4f087b26-7171-44d4-a216-7c223ebc6e4b\") " pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07004918-6af8-49a9-afc6-48bfb1fe376c-tmp\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183682    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-etc-openvswitch\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.185414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-etc-openvswitch\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-cni-dir\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-var-lib-cni-bin\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlf4\" (UniqueName: \"kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4\") pod \"network-check-target-kxk9k\" (UID: \"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610\") " pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlcxw\" (UniqueName: \"kubernetes.io/projected/07004918-6af8-49a9-afc6-48bfb1fe376c-kube-api-access-xlcxw\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.183927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-var-lib-kubelet\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-host\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-cni-dir\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-host\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-var-lib-cni-bin\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-sys-fs\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-sysctl-d\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-sysctl-conf\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.186287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-run-netns\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-run-multus-certs\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-run-multus-certs\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19401fc8-ae1c-4249-9a75-d7aab8a904de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-kubelet\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-sysctl-d\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35627b06-c338-4ef3-85b9-dd480994f542-sys-fs\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-kubelet\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-var-lib-kubelet\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-sysctl-conf\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5p66\" (UniqueName: \"kubernetes.io/projected/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-kube-api-access-j5p66\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-modprobe-d\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-cni-netd\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-run-netns\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-socket-dir-parent\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-socket-dir-parent\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bb40332-7f47-480d-8fef-7be82e907b6a-serviceca\") pod \"node-ca-t7kpc\" (UID: \"6bb40332-7f47-480d-8fef-7be82e907b6a\") " pod="openshift-image-registry/node-ca-t7kpc"
Sep 09 12:19:42.187053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19401fc8-ae1c-4249-9a75-d7aab8a904de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-cni-netd\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.184858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19401fc8-ae1c-4249-9a75-d7aab8a904de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d09d81f3-b038-48e2-88c2-6ab36ddefce2-dbus\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-cnibin\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185310    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d09d81f3-b038-48e2-88c2-6ab36ddefce2-dbus\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-cni-binary-copy\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-modprobe-d\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-conf-dir\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-cnibin\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19401fc8-ae1c-4249-9a75-d7aab8a904de-system-cni-dir\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19401fc8-ae1c-4249-9a75-d7aab8a904de-system-cni-dir\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19401fc8-ae1c-4249-9a75-d7aab8a904de-cnibin\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-slash\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-run-netns\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-env-overrides\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkdf9\" (UniqueName: \"kubernetes.io/projected/4208c0ed-c841-4b55-9475-d8816e4c3ec8-kube-api-access-tkdf9\") pod \"iptables-alerter-xh264\" (UID: \"4208c0ed-c841-4b55-9475-d8816e4c3ec8\") " pod="openshift-network-operator/iptables-alerter-xh264"
Sep 09 12:19:42.187729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d09d81f3-b038-48e2-88c2-6ab36ddefce2-kubelet-config\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-var-lib-openvswitch\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-run-openvswitch\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19401fc8-ae1c-4249-9a75-d7aab8a904de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-ovnkube-script-lib\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwbm\" (UniqueName: \"kubernetes.io/projected/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-kube-api-access-rfwbm\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp9st\" (UniqueName: \"kubernetes.io/projected/1a554455-a4e1-45a6-95d3-a129ec8da79b-kube-api-access-sp9st\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-cni-binary-copy\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.185985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-conf-dir\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19401fc8-ae1c-4249-9a75-d7aab8a904de-cnibin\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-env-overrides\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-slash\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-var-lib-openvswitch\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-host-run-netns\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-run-openvswitch\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3-tmp-dir\") pod \"node-resolver-zhgxx\" (UID: \"b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3\") " pod="openshift-dns/node-resolver-zhgxx"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-systemd\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.188198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-run\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d09d81f3-b038-48e2-88c2-6ab36ddefce2-kubelet-config\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-run-systemd\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-ovnkube-script-lib\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-system-cni-dir\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-run\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-daemon-config\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186421    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-run-systemd\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-systemd\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3-hosts-file\") pod \"node-resolver-zhgxx\" (UID: \"b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3\") " pod="openshift-dns/node-resolver-zhgxx"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-sysconfig\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-etc-kubernetes\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-sysconfig\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-system-cni-dir\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-kubernetes\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-var-lib-kubelet\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-kubernetes\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.188804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8klc9\" (UniqueName: \"kubernetes.io/projected/6bb40332-7f47-480d-8fef-7be82e907b6a-kube-api-access-8klc9\") pod \"node-ca-t7kpc\" (UID: \"6bb40332-7f47-480d-8fef-7be82e907b6a\") " pod="openshift-image-registry/node-ca-t7kpc"
Sep 09 12:19:42.189435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4208c0ed-c841-4b55-9475-d8816e4c3ec8-host-slash\") pod \"iptables-alerter-xh264\" (UID: \"4208c0ed-c841-4b55-9475-d8816e4c3ec8\") " pod="openshift-network-operator/iptables-alerter-xh264"
Sep 09 12:19:42.189435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-host-var-lib-kubelet\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.189435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvc5\" (UniqueName: \"kubernetes.io/projected/35627b06-c338-4ef3-85b9-dd480994f542-kube-api-access-tzvc5\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.189435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-etc-kubernetes\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.189435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.186863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-multus-daemon-config\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.189435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.187372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07004918-6af8-49a9-afc6-48bfb1fe376c-tmp\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.189435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.187459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07004918-6af8-49a9-afc6-48bfb1fe376c-etc-tuned\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.189435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.187623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-ovn-node-metrics-cert\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.189435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.189355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal" event={"ID":"aea0489aa54ab9ac328abed7fe01c8c4","Type":"ContainerStarted","Data":"70b9f73d3ba39fcb510e2d86a4d7cd9c1fe9e544ef9d255b09aaa4f1ccdff3f3"}
Sep 09 12:19:42.191403 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.191375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-152.ec2.internal" event={"ID":"a3308eb29717fa7fe5c5ea5d97019ea1","Type":"ContainerStarted","Data":"bc7959ee97a4b4dbddc358ce61c4b6e01e9b3db0322d8160de372a540f75a45b"}
Sep 09 12:19:42.193461 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.193439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7kn\" (UniqueName: \"kubernetes.io/projected/19401fc8-ae1c-4249-9a75-d7aab8a904de-kube-api-access-rc7kn\") pod \"multus-additional-cni-plugins-nvm9j\" (UID: \"19401fc8-ae1c-4249-9a75-d7aab8a904de\") " pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.199963 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.199940    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:19:42.199963 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.199963    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:19:42.200130 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.199974    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8jlf4 for pod openshift-network-diagnostics/network-check-target-kxk9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:42.200130 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.200022    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4 podName:004699ec-e8f0-4fb3-87b1-0d5a4ff3a610 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:42.700009628 +0000 UTC m=+3.043607554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlf4" (UniqueName: "kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4") pod "network-check-target-kxk9k" (UID: "004699ec-e8f0-4fb3-87b1-0d5a4ff3a610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:42.202677 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.202652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5p66\" (UniqueName: \"kubernetes.io/projected/6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113-kube-api-access-j5p66\") pod \"multus-kc57k\" (UID: \"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113\") " pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.202793 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.202779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwbm\" (UniqueName: \"kubernetes.io/projected/5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9-kube-api-access-rfwbm\") pod \"ovnkube-node-4nvnq\" (UID: \"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.202973 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.202956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvc5\" (UniqueName: \"kubernetes.io/projected/35627b06-c338-4ef3-85b9-dd480994f542-kube-api-access-tzvc5\") pod \"aws-ebs-csi-driver-node-6vdt2\" (UID: \"35627b06-c338-4ef3-85b9-dd480994f542\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.203119 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.203104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlcxw\" (UniqueName: \"kubernetes.io/projected/07004918-6af8-49a9-afc6-48bfb1fe376c-kube-api-access-xlcxw\") pod \"tuned-cwjkj\" (UID: \"07004918-6af8-49a9-afc6-48bfb1fe376c\") " pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.287276 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f087b26-7171-44d4-a216-7c223ebc6e4b-konnectivity-ca\") pod \"konnectivity-agent-jf5dm\" (UID: \"4f087b26-7171-44d4-a216-7c223ebc6e4b\") " pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:19:42.287276 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bb40332-7f47-480d-8fef-7be82e907b6a-host\") pod \"node-ca-t7kpc\" (UID: \"6bb40332-7f47-480d-8fef-7be82e907b6a\") " pod="openshift-image-registry/node-ca-t7kpc"
Sep 09 12:19:42.287276 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmt57\" (UniqueName: \"kubernetes.io/projected/b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3-kube-api-access-lmt57\") pod \"node-resolver-zhgxx\" (UID: \"b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3\") " pod="openshift-dns/node-resolver-zhgxx"
Sep 09 12:19:42.287547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4208c0ed-c841-4b55-9475-d8816e4c3ec8-iptables-alerter-script\") pod \"iptables-alerter-xh264\" (UID: \"4208c0ed-c841-4b55-9475-d8816e4c3ec8\") " pod="openshift-network-operator/iptables-alerter-xh264"
Sep 09 12:19:42.287547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f087b26-7171-44d4-a216-7c223ebc6e4b-agent-certs\") pod \"konnectivity-agent-jf5dm\" (UID: \"4f087b26-7171-44d4-a216-7c223ebc6e4b\") " pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:19:42.287547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bb40332-7f47-480d-8fef-7be82e907b6a-serviceca\") pod \"node-ca-t7kpc\" (UID: \"6bb40332-7f47-480d-8fef-7be82e907b6a\") " pod="openshift-image-registry/node-ca-t7kpc"
Sep 09 12:19:42.287547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bb40332-7f47-480d-8fef-7be82e907b6a-host\") pod \"node-ca-t7kpc\" (UID: \"6bb40332-7f47-480d-8fef-7be82e907b6a\") " pod="openshift-image-registry/node-ca-t7kpc"
Sep 09 12:19:42.287547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkdf9\" (UniqueName: \"kubernetes.io/projected/4208c0ed-c841-4b55-9475-d8816e4c3ec8-kube-api-access-tkdf9\") pod \"iptables-alerter-xh264\" (UID: \"4208c0ed-c841-4b55-9475-d8816e4c3ec8\") " pod="openshift-network-operator/iptables-alerter-xh264"
Sep 09 12:19:42.287547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp9st\" (UniqueName: \"kubernetes.io/projected/1a554455-a4e1-45a6-95d3-a129ec8da79b-kube-api-access-sp9st\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:42.287547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3-tmp-dir\") pod \"node-resolver-zhgxx\" (UID: \"b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3\") " pod="openshift-dns/node-resolver-zhgxx"
Sep 09 12:19:42.287547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3-hosts-file\") pod \"node-resolver-zhgxx\" (UID: \"b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3\") " pod="openshift-dns/node-resolver-zhgxx"
Sep 09 12:19:42.287957 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:42.287957 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8klc9\" (UniqueName: \"kubernetes.io/projected/6bb40332-7f47-480d-8fef-7be82e907b6a-kube-api-access-8klc9\") pod \"node-ca-t7kpc\" (UID: \"6bb40332-7f47-480d-8fef-7be82e907b6a\") " pod="openshift-image-registry/node-ca-t7kpc"
Sep 09 12:19:42.287957 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4208c0ed-c841-4b55-9475-d8816e4c3ec8-host-slash\") pod \"iptables-alerter-xh264\" (UID: \"4208c0ed-c841-4b55-9475-d8816e4c3ec8\") " pod="openshift-network-operator/iptables-alerter-xh264"
Sep 09 12:19:42.287957 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4208c0ed-c841-4b55-9475-d8816e4c3ec8-host-slash\") pod \"iptables-alerter-xh264\" (UID: \"4208c0ed-c841-4b55-9475-d8816e4c3ec8\") " pod="openshift-network-operator/iptables-alerter-xh264"
Sep 09 12:19:42.287957 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4208c0ed-c841-4b55-9475-d8816e4c3ec8-iptables-alerter-script\") pod \"iptables-alerter-xh264\" (UID: \"4208c0ed-c841-4b55-9475-d8816e4c3ec8\") " pod="openshift-network-operator/iptables-alerter-xh264"
Sep 09 12:19:42.287957 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f087b26-7171-44d4-a216-7c223ebc6e4b-konnectivity-ca\") pod \"konnectivity-agent-jf5dm\" (UID: \"4f087b26-7171-44d4-a216-7c223ebc6e4b\") " pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:19:42.287957 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.287877    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3-hosts-file\") pod \"node-resolver-zhgxx\" (UID: \"b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3\") " pod="openshift-dns/node-resolver-zhgxx"
Sep 09 12:19:42.287957 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.287928    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:42.288319 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.287989    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs podName:1a554455-a4e1-45a6-95d3-a129ec8da79b nodeName:}" failed. No retries permitted until 2026-09-09 12:19:42.787977177 +0000 UTC m=+3.131575102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs") pod "network-metrics-daemon-thh9h" (UID: "1a554455-a4e1-45a6-95d3-a129ec8da79b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:42.288319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.288112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3-tmp-dir\") pod \"node-resolver-zhgxx\" (UID: \"b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3\") " pod="openshift-dns/node-resolver-zhgxx"
Sep 09 12:19:42.288708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.288692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bb40332-7f47-480d-8fef-7be82e907b6a-serviceca\") pod \"node-ca-t7kpc\" (UID: \"6bb40332-7f47-480d-8fef-7be82e907b6a\") " pod="openshift-image-registry/node-ca-t7kpc"
Sep 09 12:19:42.290532 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.290514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f087b26-7171-44d4-a216-7c223ebc6e4b-agent-certs\") pod \"konnectivity-agent-jf5dm\" (UID: \"4f087b26-7171-44d4-a216-7c223ebc6e4b\") " pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:19:42.296085 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.296061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp9st\" (UniqueName: \"kubernetes.io/projected/1a554455-a4e1-45a6-95d3-a129ec8da79b-kube-api-access-sp9st\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:42.296496 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.296479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkdf9\" (UniqueName: \"kubernetes.io/projected/4208c0ed-c841-4b55-9475-d8816e4c3ec8-kube-api-access-tkdf9\") pod \"iptables-alerter-xh264\" (UID: \"4208c0ed-c841-4b55-9475-d8816e4c3ec8\") " pod="openshift-network-operator/iptables-alerter-xh264"
Sep 09 12:19:42.297174 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.297154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmt57\" (UniqueName: \"kubernetes.io/projected/b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3-kube-api-access-lmt57\") pod \"node-resolver-zhgxx\" (UID: \"b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3\") " pod="openshift-dns/node-resolver-zhgxx"
Sep 09 12:19:42.297543 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.297518    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8klc9\" (UniqueName: \"kubernetes.io/projected/6bb40332-7f47-480d-8fef-7be82e907b6a-kube-api-access-8klc9\") pod \"node-ca-t7kpc\" (UID: \"6bb40332-7f47-480d-8fef-7be82e907b6a\") " pod="openshift-image-registry/node-ca-t7kpc"
Sep 09 12:19:42.372572 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.372538    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:19:42.383006 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.382984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kc57k"
Sep 09 12:19:42.391836 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.391812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2"
Sep 09 12:19:42.397412 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.397391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:19:42.404013 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.403995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cwjkj"
Sep 09 12:19:42.410594 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.410567    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nvm9j"
Sep 09 12:19:42.417088 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.417072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t7kpc"
Sep 09 12:19:42.424781 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.424759    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xh264"
Sep 09 12:19:42.429365 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.429346    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zhgxx"
Sep 09 12:19:42.594423 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:42.594379    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b7fd24_9e5e_488e_aa9a_62a5531e9ab3.slice/crio-2a53496bf6542a14955bb4a9c210a98c3e2e508d996c9ecf832f7ad2ad4bc71d WatchSource:0}: Error finding container 2a53496bf6542a14955bb4a9c210a98c3e2e508d996c9ecf832f7ad2ad4bc71d: Status 404 returned error can't find the container with id 2a53496bf6542a14955bb4a9c210a98c3e2e508d996c9ecf832f7ad2ad4bc71d
Sep 09 12:19:42.595661 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:42.595174    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab00a7c_2dc0_49df_b9ac_1ad93d7fc113.slice/crio-fb75fc403ab002e5ee55d60b181a5a7c2acb647719128db161c5e747c5ee582c WatchSource:0}: Error finding container fb75fc403ab002e5ee55d60b181a5a7c2acb647719128db161c5e747c5ee582c: Status 404 returned error can't find the container with id fb75fc403ab002e5ee55d60b181a5a7c2acb647719128db161c5e747c5ee582c
Sep 09 12:19:42.599384 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:42.599361    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07004918_6af8_49a9_afc6_48bfb1fe376c.slice/crio-ea77853611de76f6c81fdfc76cea45b12e42cd82a6420bf1c371a7a46b01544c WatchSource:0}: Error finding container ea77853611de76f6c81fdfc76cea45b12e42cd82a6420bf1c371a7a46b01544c: Status 404 returned error can't find the container with id ea77853611de76f6c81fdfc76cea45b12e42cd82a6420bf1c371a7a46b01544c
Sep 09 12:19:42.601480 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:42.601460    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c71bfd3_9aa6_4e71_b7ec_6e86e54022d9.slice/crio-847e691df6658005db17d7fd86d19e386b93be229bebbe7bec2e50ae495bb556 WatchSource:0}: Error finding container 847e691df6658005db17d7fd86d19e386b93be229bebbe7bec2e50ae495bb556: Status 404 returned error can't find the container with id 847e691df6658005db17d7fd86d19e386b93be229bebbe7bec2e50ae495bb556
Sep 09 12:19:42.620979 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:42.620954    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19401fc8_ae1c_4249_9a75_d7aab8a904de.slice/crio-6ebd67bc857a3a551dd341c9b7556849898be8b800fb70dfae8d86d2cdd7cab0 WatchSource:0}: Error finding container 6ebd67bc857a3a551dd341c9b7556849898be8b800fb70dfae8d86d2cdd7cab0: Status 404 returned error can't find the container with id 6ebd67bc857a3a551dd341c9b7556849898be8b800fb70dfae8d86d2cdd7cab0
Sep 09 12:19:42.621646 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:42.621626    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35627b06_c338_4ef3_85b9_dd480994f542.slice/crio-b244f64f06f452ed9d88ace580f5390a94e141fd9a57cb98bb41dfef4f448a3b WatchSource:0}: Error finding container b244f64f06f452ed9d88ace580f5390a94e141fd9a57cb98bb41dfef4f448a3b: Status 404 returned error can't find the container with id b244f64f06f452ed9d88ace580f5390a94e141fd9a57cb98bb41dfef4f448a3b
Sep 09 12:19:42.622565 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:42.622389    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4208c0ed_c841_4b55_9475_d8816e4c3ec8.slice/crio-98496ce229b6cd0ab54d27d89308dfe0b7665185bd54722e5e37bb5d8dae8e7f WatchSource:0}: Error finding container 98496ce229b6cd0ab54d27d89308dfe0b7665185bd54722e5e37bb5d8dae8e7f: Status 404 returned error can't find the container with id 98496ce229b6cd0ab54d27d89308dfe0b7665185bd54722e5e37bb5d8dae8e7f
Sep 09 12:19:42.623513 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:42.623495    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb40332_7f47_480d_8fef_7be82e907b6a.slice/crio-c9020ee1d3a8afadb2b8633da1b8156b3d3395654867dbceed92c4f778b7efff WatchSource:0}: Error finding container c9020ee1d3a8afadb2b8633da1b8156b3d3395654867dbceed92c4f778b7efff: Status 404 returned error can't find the container with id c9020ee1d3a8afadb2b8633da1b8156b3d3395654867dbceed92c4f778b7efff
Sep 09 12:19:42.624023 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:19:42.624003    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f087b26_7171_44d4_a216_7c223ebc6e4b.slice/crio-a508879a0dd3d8a57b79876115b35280634fc50648beb46ca42c893afc066e77 WatchSource:0}: Error finding container a508879a0dd3d8a57b79876115b35280634fc50648beb46ca42c893afc066e77: Status 404 returned error can't find the container with id a508879a0dd3d8a57b79876115b35280634fc50648beb46ca42c893afc066e77
Sep 09 12:19:42.691001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.690967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:42.691160 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.691109    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:42.691240 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.691180    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret podName:d09d81f3-b038-48e2-88c2-6ab36ddefce2 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:43.691161647 +0000 UTC m=+4.034759590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret") pod "global-pull-secret-syncer-qdgrt" (UID: "d09d81f3-b038-48e2-88c2-6ab36ddefce2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:42.791980 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.791946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:42.792128 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:42.792043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlf4\" (UniqueName: \"kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4\") pod \"network-check-target-kxk9k\" (UID: \"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610\") " pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:42.792128 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.792096    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:42.792241 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.792174    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs podName:1a554455-a4e1-45a6-95d3-a129ec8da79b nodeName:}" failed. No retries permitted until 2026-09-09 12:19:43.792153273 +0000 UTC m=+4.135751201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs") pod "network-metrics-daemon-thh9h" (UID: "1a554455-a4e1-45a6-95d3-a129ec8da79b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:42.792241 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.792197    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:19:42.792322 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.792245    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:19:42.792322 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.792261    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8jlf4 for pod openshift-network-diagnostics/network-check-target-kxk9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:42.792400 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:42.792331    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4 podName:004699ec-e8f0-4fb3-87b1-0d5a4ff3a610 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:43.792315626 +0000 UTC m=+4.135913555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlf4" (UniqueName: "kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4") pod "network-check-target-kxk9k" (UID: "004699ec-e8f0-4fb3-87b1-0d5a4ff3a610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:43.115789 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.115745    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 12:14:42 +0000 UTC" deadline="2028-06-22 19:28:01.683687712 +0000 UTC"
Sep 09 12:19:43.115789 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.115782    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15655h8m18.567908892s"
Sep 09 12:19:43.184388 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.183770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:43.184388 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:43.183864    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:19:43.184388 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.184269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:43.184388 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:43.184345    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:19:43.197581 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.197544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t7kpc" event={"ID":"6bb40332-7f47-480d-8fef-7be82e907b6a","Type":"ContainerStarted","Data":"c9020ee1d3a8afadb2b8633da1b8156b3d3395654867dbceed92c4f778b7efff"}
Sep 09 12:19:43.199898 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.199870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2" event={"ID":"35627b06-c338-4ef3-85b9-dd480994f542","Type":"ContainerStarted","Data":"b244f64f06f452ed9d88ace580f5390a94e141fd9a57cb98bb41dfef4f448a3b"}
Sep 09 12:19:43.202071 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.202021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nvm9j" event={"ID":"19401fc8-ae1c-4249-9a75-d7aab8a904de","Type":"ContainerStarted","Data":"6ebd67bc857a3a551dd341c9b7556849898be8b800fb70dfae8d86d2cdd7cab0"}
Sep 09 12:19:43.205120 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.205093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cwjkj" event={"ID":"07004918-6af8-49a9-afc6-48bfb1fe376c","Type":"ContainerStarted","Data":"ea77853611de76f6c81fdfc76cea45b12e42cd82a6420bf1c371a7a46b01544c"}
Sep 09 12:19:43.208925 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.208888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq" event={"ID":"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9","Type":"ContainerStarted","Data":"847e691df6658005db17d7fd86d19e386b93be229bebbe7bec2e50ae495bb556"}
Sep 09 12:19:43.210401 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.210355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zhgxx" event={"ID":"b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3","Type":"ContainerStarted","Data":"2a53496bf6542a14955bb4a9c210a98c3e2e508d996c9ecf832f7ad2ad4bc71d"}
Sep 09 12:19:43.216812 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.216783    2578 generic.go:356] "Generic (PLEG): container finished" podID="aea0489aa54ab9ac328abed7fe01c8c4" containerID="a095c44c8a40f561f6b059570cab5a79a721c68c6057ebe596123ebd0b78a209" exitCode=0
Sep 09 12:19:43.216902 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.216864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal" event={"ID":"aea0489aa54ab9ac328abed7fe01c8c4","Type":"ContainerDied","Data":"a095c44c8a40f561f6b059570cab5a79a721c68c6057ebe596123ebd0b78a209"}
Sep 09 12:19:43.227747 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.227628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jf5dm" event={"ID":"4f087b26-7171-44d4-a216-7c223ebc6e4b","Type":"ContainerStarted","Data":"a508879a0dd3d8a57b79876115b35280634fc50648beb46ca42c893afc066e77"}
Sep 09 12:19:43.233594 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.233528    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xh264" event={"ID":"4208c0ed-c841-4b55-9475-d8816e4c3ec8","Type":"ContainerStarted","Data":"98496ce229b6cd0ab54d27d89308dfe0b7665185bd54722e5e37bb5d8dae8e7f"}
Sep 09 12:19:43.236007 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.235966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kc57k" event={"ID":"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113","Type":"ContainerStarted","Data":"fb75fc403ab002e5ee55d60b181a5a7c2acb647719128db161c5e747c5ee582c"}
Sep 09 12:19:43.700245 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.699850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:43.700245 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:43.700111    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:43.700245 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:43.700198    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret podName:d09d81f3-b038-48e2-88c2-6ab36ddefce2 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:45.700177988 +0000 UTC m=+6.043775919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret") pod "global-pull-secret-syncer-qdgrt" (UID: "d09d81f3-b038-48e2-88c2-6ab36ddefce2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:43.800785 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.800746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlf4\" (UniqueName: \"kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4\") pod \"network-check-target-kxk9k\" (UID: \"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610\") " pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:43.800976 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:43.800832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:43.801035 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:43.800994    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:43.801088 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:43.801058    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs podName:1a554455-a4e1-45a6-95d3-a129ec8da79b nodeName:}" failed. No retries permitted until 2026-09-09 12:19:45.801040286 +0000 UTC m=+6.144638228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs") pod "network-metrics-daemon-thh9h" (UID: "1a554455-a4e1-45a6-95d3-a129ec8da79b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:43.801500 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:43.801479    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:19:43.801593 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:43.801505    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:19:43.801593 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:43.801517    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8jlf4 for pod openshift-network-diagnostics/network-check-target-kxk9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:43.801593 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:43.801561    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4 podName:004699ec-e8f0-4fb3-87b1-0d5a4ff3a610 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:45.801547058 +0000 UTC m=+6.145144987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlf4" (UniqueName: "kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4") pod "network-check-target-kxk9k" (UID: "004699ec-e8f0-4fb3-87b1-0d5a4ff3a610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:44.116144 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:44.116076    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 12:14:42 +0000 UTC" deadline="2028-02-08 00:56:14.205760689 +0000 UTC"
Sep 09 12:19:44.116144 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:44.116141    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12396h36m30.089623592s"
Sep 09 12:19:44.181408 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:44.181375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:44.181586 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:44.181523    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:19:44.261293 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:44.261190    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal" event={"ID":"aea0489aa54ab9ac328abed7fe01c8c4","Type":"ContainerStarted","Data":"477eb8232ad0f2e7e50675c5383364e48c9446d130c05dc2a54359bb727ffae3"}
Sep 09 12:19:45.180283 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:45.180250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:45.180749 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:45.180386    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:19:45.180749 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:45.180250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:45.180871 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:45.180806    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:19:45.722651 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:45.721915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:45.722651 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:45.722189    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:45.722651 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:45.722274    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret podName:d09d81f3-b038-48e2-88c2-6ab36ddefce2 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:49.722255914 +0000 UTC m=+10.065853853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret") pod "global-pull-secret-syncer-qdgrt" (UID: "d09d81f3-b038-48e2-88c2-6ab36ddefce2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:45.823515 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:45.822665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:45.823515 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:45.822740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlf4\" (UniqueName: \"kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4\") pod \"network-check-target-kxk9k\" (UID: \"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610\") " pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:45.823515 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:45.822909    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:19:45.823515 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:45.822933    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:19:45.823515 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:45.822945    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8jlf4 for pod openshift-network-diagnostics/network-check-target-kxk9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:45.823515 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:45.823000    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4 podName:004699ec-e8f0-4fb3-87b1-0d5a4ff3a610 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:49.822980831 +0000 UTC m=+10.166578761 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlf4" (UniqueName: "kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4") pod "network-check-target-kxk9k" (UID: "004699ec-e8f0-4fb3-87b1-0d5a4ff3a610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:45.823515 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:45.823417    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:45.823515 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:45.823476    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs podName:1a554455-a4e1-45a6-95d3-a129ec8da79b nodeName:}" failed. No retries permitted until 2026-09-09 12:19:49.823460995 +0000 UTC m=+10.167058924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs") pod "network-metrics-daemon-thh9h" (UID: "1a554455-a4e1-45a6-95d3-a129ec8da79b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:46.181755 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:46.180473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:46.181755 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:46.181420    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:19:47.180283 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:47.180248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:47.180506 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:47.180390    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:19:47.180715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:47.180250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:47.180822 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:47.180800    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:19:48.180646 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:48.180611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:48.181152 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:48.180723    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:19:49.179868 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:49.179737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:49.179868 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:49.179767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:49.180085 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:49.179874    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:19:49.180085 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:49.180001    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:19:49.758478 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:49.758357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:49.758936 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:49.758529    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:49.758936 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:49.758605    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret podName:d09d81f3-b038-48e2-88c2-6ab36ddefce2 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:57.758585516 +0000 UTC m=+18.102183454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret") pod "global-pull-secret-syncer-qdgrt" (UID: "d09d81f3-b038-48e2-88c2-6ab36ddefce2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:49.859598 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:49.859564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlf4\" (UniqueName: \"kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4\") pod \"network-check-target-kxk9k\" (UID: \"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610\") " pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:49.859772 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:49.859622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:49.859772 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:49.859690    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:19:49.859772 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:49.859714    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:19:49.859772 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:49.859717    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:49.859772 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:49.859725    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8jlf4 for pod openshift-network-diagnostics/network-check-target-kxk9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:49.859772 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:49.859771    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs podName:1a554455-a4e1-45a6-95d3-a129ec8da79b nodeName:}" failed. No retries permitted until 2026-09-09 12:19:57.859752591 +0000 UTC m=+18.203350517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs") pod "network-metrics-daemon-thh9h" (UID: "1a554455-a4e1-45a6-95d3-a129ec8da79b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:49.860110 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:49.859790    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4 podName:004699ec-e8f0-4fb3-87b1-0d5a4ff3a610 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:57.859780271 +0000 UTC m=+18.203378199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlf4" (UniqueName: "kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4") pod "network-check-target-kxk9k" (UID: "004699ec-e8f0-4fb3-87b1-0d5a4ff3a610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:50.181108 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:50.180639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:50.181108 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:50.180764    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:19:51.180283 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:51.179626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:51.180283 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:51.179755    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:19:51.180283 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:51.180129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:51.180283 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:51.180236    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:19:52.180051 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:52.180016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:52.180228 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:52.180147    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:19:53.179626 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:53.179597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:53.179960 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:53.179597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:53.179960 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:53.179704    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:19:53.179960 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:53.179772    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:19:54.179967 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:54.179937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:54.180425 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:54.180086    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:19:55.180476 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:55.180444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:55.180867 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:55.180445    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:55.180867 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:55.180579    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:19:55.180867 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:55.180654    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:19:56.180504 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:56.180461    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:56.180969 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:56.180618    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:19:57.179982 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:57.179949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:57.180175 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:57.179947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:57.180175 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:57.180084    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:19:57.180175 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:57.180113    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:19:57.820547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:57.820508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:57.821000 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:57.820650    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:57.821000 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:57.820711    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret podName:d09d81f3-b038-48e2-88c2-6ab36ddefce2 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:13.820695825 +0000 UTC m=+34.164293750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret") pod "global-pull-secret-syncer-qdgrt" (UID: "d09d81f3-b038-48e2-88c2-6ab36ddefce2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:19:57.921650 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:57.921611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:57.921837 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:57.921692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlf4\" (UniqueName: \"kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4\") pod \"network-check-target-kxk9k\" (UID: \"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610\") " pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:57.921837 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:57.921806    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:57.921934 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:57.921834    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:19:57.921934 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:57.921857    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:19:57.921934 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:57.921871    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8jlf4 for pod openshift-network-diagnostics/network-check-target-kxk9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:57.921934 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:57.921888    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs podName:1a554455-a4e1-45a6-95d3-a129ec8da79b nodeName:}" failed. No retries permitted until 2026-09-09 12:20:13.921866413 +0000 UTC m=+34.265464349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs") pod "network-metrics-daemon-thh9h" (UID: "1a554455-a4e1-45a6-95d3-a129ec8da79b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:19:57.921934 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:57.921923    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4 podName:004699ec-e8f0-4fb3-87b1-0d5a4ff3a610 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:13.921907084 +0000 UTC m=+34.265505066 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlf4" (UniqueName: "kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4") pod "network-check-target-kxk9k" (UID: "004699ec-e8f0-4fb3-87b1-0d5a4ff3a610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:19:58.179892 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:58.179863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:19:58.180071 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:58.180002    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:19:59.180389 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:59.180358    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:19:59.180818 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:19:59.180355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:19:59.180818 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:59.180491    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:19:59.180818 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:19:59.180558    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:20:00.183168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:00.181413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:00.183168 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:00.181565    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:20:01.179995 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:01.179961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:01.180180 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:01.179962    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:01.180180 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:01.180062    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:20:01.180180 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:01.180154    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:20:02.180394 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.180066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:02.181179 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:02.180503    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:20:02.300996 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.300963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kc57k" event={"ID":"6ab00a7c-2dc0-49df-b9ac-1ad93d7fc113","Type":"ContainerStarted","Data":"67014d31ca68550f16162021546248811b0cb4befda47962ba7523b83ce0e641"}
Sep 09 12:20:02.302543 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.302519    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t7kpc" event={"ID":"6bb40332-7f47-480d-8fef-7be82e907b6a","Type":"ContainerStarted","Data":"019fe65528feaa7739913a7cdb85dff2bbdf8fe12584abe544cac036df6d53f9"}
Sep 09 12:20:02.304167 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.304132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2" event={"ID":"35627b06-c338-4ef3-85b9-dd480994f542","Type":"ContainerStarted","Data":"aaec777ffb7287fdbf257152b8e92a91dd1f531c0dfe3587a6661e87d8a19338"}
Sep 09 12:20:02.305615 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.305593    2578 generic.go:356] "Generic (PLEG): container finished" podID="19401fc8-ae1c-4249-9a75-d7aab8a904de" containerID="83944e317fdb49f2e14954be7bce043895c4dd1f5894a16012999929be0bfa1b" exitCode=0
Sep 09 12:20:02.305704 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.305666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nvm9j" event={"ID":"19401fc8-ae1c-4249-9a75-d7aab8a904de","Type":"ContainerDied","Data":"83944e317fdb49f2e14954be7bce043895c4dd1f5894a16012999929be0bfa1b"}
Sep 09 12:20:02.307276 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.307209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cwjkj" event={"ID":"07004918-6af8-49a9-afc6-48bfb1fe376c","Type":"ContainerStarted","Data":"79e47015dedb340da0ced70576fc24946239097b0d5354eebb9c95164f586e86"}
Sep 09 12:20:02.312428 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.312407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq" event={"ID":"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9","Type":"ContainerStarted","Data":"673cf092ac1f000a7f9a7e2a2494571ae5f68010b3f3f273ccc9a7ccfe6dd00c"}
Sep 09 12:20:02.312511 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.312474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq" event={"ID":"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9","Type":"ContainerStarted","Data":"d661e3602d3cd702477d5488f9c24042834f2a7e7473cff718175963d1d1bfe4"}
Sep 09 12:20:02.312554 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.312518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq" event={"ID":"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9","Type":"ContainerStarted","Data":"d64cb616eaaf39e6a4641b88e150fedc6362f52d7a3af9c46a2bd1c7b1b9f88d"}
Sep 09 12:20:02.312600 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.312552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq" event={"ID":"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9","Type":"ContainerStarted","Data":"f076fb06b5168349107112aed6dfe1e7bbd1f78f2ade5fd8b2f8f35fbe136a66"}
Sep 09 12:20:02.312600 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.312579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq" event={"ID":"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9","Type":"ContainerStarted","Data":"9ffb17f78ea8215d487cf2f35e5b2bc407dbe08509ad6170169663691cd1d16a"}
Sep 09 12:20:02.312678 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.312612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq" event={"ID":"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9","Type":"ContainerStarted","Data":"020a7a1f17c0d5a549fd9af6c9fbdb93ab9eafe9c01438e0142176118d4987f1"}
Sep 09 12:20:02.313754 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.313735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zhgxx" event={"ID":"b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3","Type":"ContainerStarted","Data":"4075c60296be074989f469627e5d84b0ceee9433c1eaacdec7b40b555fdf6dea"}
Sep 09 12:20:02.315143 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.315122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-152.ec2.internal" event={"ID":"a3308eb29717fa7fe5c5ea5d97019ea1","Type":"ContainerStarted","Data":"26d2241f7879b3487483624ce92853c042f94ea7603961794f9d2f21ca9956e4"}
Sep 09 12:20:02.316400 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.316382    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jf5dm" event={"ID":"4f087b26-7171-44d4-a216-7c223ebc6e4b","Type":"ContainerStarted","Data":"2ba3febc437f9343a6c421f53e379049a4e6174d5545da1e28814e7e96fc86ee"}
Sep 09 12:20:02.317766 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.317729    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-152.ec2.internal" podStartSLOduration=21.317717896 podStartE2EDuration="21.317717896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:19:44.278451903 +0000 UTC m=+4.622049854" watchObservedRunningTime="2026-09-09 12:20:02.317717896 +0000 UTC m=+22.661315844"
Sep 09 12:20:02.317875 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.317816    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kc57k" podStartSLOduration=3.1912627000000002 podStartE2EDuration="22.317808047s" totalImagesPullingTime="19.126545347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:02.31750906 +0000 UTC m=+22.661107008" watchObservedRunningTime="2026-09-09 12:20:02.317808047 +0000 UTC m=+22.661405997"
Sep 09 12:20:02.333947 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.333900    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zhgxx" podStartSLOduration=3.538082526 podStartE2EDuration="22.333891708s" totalImagesPullingTime="18.795809182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:02.333715789 +0000 UTC m=+22.677313729" watchObservedRunningTime="2026-09-09 12:20:02.333891708 +0000 UTC m=+22.677489656"
Sep 09 12:20:02.347036 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.346992    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-152.ec2.internal" podStartSLOduration=21.346979429 podStartE2EDuration="21.346979429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:02.346787555 +0000 UTC m=+22.690385503" watchObservedRunningTime="2026-09-09 12:20:02.346979429 +0000 UTC m=+22.690577378"
Sep 09 12:20:02.361122 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.361086    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t7kpc" podStartSLOduration=3.509277619 podStartE2EDuration="22.361072337s" totalImagesPullingTime="18.851794718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:02.3607853 +0000 UTC m=+22.704383247" watchObservedRunningTime="2026-09-09 12:20:02.361072337 +0000 UTC m=+22.704670284"
Sep 09 12:20:02.373665 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.373631    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jf5dm" podStartSLOduration=11.31718312 podStartE2EDuration="22.373614508s" totalImagesPullingTime="11.056431388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:02.373602935 +0000 UTC m=+22.717200883" watchObservedRunningTime="2026-09-09 12:20:02.373614508 +0000 UTC m=+22.717212455"
Sep 09 12:20:02.393651 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:02.393609    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cwjkj" podStartSLOduration=3.533456383 podStartE2EDuration="22.39359616s" totalImagesPullingTime="18.860139777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:02.393204446 +0000 UTC m=+22.736802391" watchObservedRunningTime="2026-09-09 12:20:02.39359616 +0000 UTC m=+22.737194107"
Sep 09 12:20:03.180568 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:03.180536    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:03.180975 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:03.180544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:03.180975 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:03.180706    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:20:03.180975 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:03.180743    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:20:03.255037 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:03.255006    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:20:03.320968 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:03.320932    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xh264" event={"ID":"4208c0ed-c841-4b55-9475-d8816e4c3ec8","Type":"ContainerStarted","Data":"8b90b0abc58d1959c81621490b35c547bb7f290953cfd0c561db46602f987b46"}
Sep 09 12:20:03.335605 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:03.335561    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xh264" podStartSLOduration=4.506156569 podStartE2EDuration="23.335544234s" totalImagesPullingTime="18.829387665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:03.335201984 +0000 UTC m=+23.678799932" watchObservedRunningTime="2026-09-09 12:20:03.335544234 +0000 UTC m=+23.679142188"
Sep 09 12:20:03.510863 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:03.510832    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 12:20:04.153666 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:04.153565    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T12:20:03.510862599Z","UUID":"e176b0cd-2e0c-48a3-97ff-bea2cc5cebb0","Handler":null,"Name":"","Endpoint":""}
Sep 09 12:20:04.158125 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:04.158100    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 12:20:04.158288 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:04.158141    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 12:20:04.179732 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:04.179704    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:04.179883 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:04.179831    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:20:04.325292 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:04.325255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2" event={"ID":"35627b06-c338-4ef3-85b9-dd480994f542","Type":"ContainerStarted","Data":"0a76f1423e29adc98e59109ca9651f46a56a33b7b21c921975ed7294c5c1d555"}
Sep 09 12:20:04.329434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:04.329355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq" event={"ID":"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9","Type":"ContainerStarted","Data":"d5a40b6063c155ff21b01a12edbe1df5c0a949d01caad03057f08e387158059b"}
Sep 09 12:20:05.180126 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:05.180087    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:05.180324 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:05.180136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:05.180324 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:05.180194    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:20:05.180324 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:05.180309    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:20:05.335036 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:05.334953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2" event={"ID":"35627b06-c338-4ef3-85b9-dd480994f542","Type":"ContainerStarted","Data":"e151417ece03d923d02adde77adbb8597b727d2f09c9bd88f2ebf19850b85ce5"}
Sep 09 12:20:05.352466 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:05.352423    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vdt2" podStartSLOduration=3.339172762 podStartE2EDuration="25.3524052s" totalImagesPullingTime="22.013232438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:05.352363267 +0000 UTC m=+25.695961215" watchObservedRunningTime="2026-09-09 12:20:05.3524052 +0000 UTC m=+25.696003142"
Sep 09 12:20:06.180166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:06.180002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:06.180292 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:06.180273    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:20:06.876090 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:06.876054    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:20:06.876905 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:06.876882    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:20:07.179699 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:07.179672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:07.179793 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:07.179777    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:20:07.179835 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:07.179826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:07.179922 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:07.179907    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:20:07.344694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:07.344646    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq" event={"ID":"5c71bfd3-9aa6-4e71-b7ec-6e86e54022d9","Type":"ContainerStarted","Data":"2248c4f94babbb67479b9b8f78e0f8dc7dc1666167be7b5c24b2b2f7b784f47e"}
Sep 09 12:20:07.345642 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:07.345619    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jf5dm"
Sep 09 12:20:07.394317 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:07.394267    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq" podStartSLOduration=8.519829044 podStartE2EDuration="27.394247513s" totalImagesPullingTime="18.874418469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:07.374247487 +0000 UTC m=+27.717845434" watchObservedRunningTime="2026-09-09 12:20:07.394247513 +0000 UTC m=+27.737845459"
Sep 09 12:20:08.180378 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:08.180345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:08.181091 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:08.180455    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:20:08.349315 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:08.349189    2578 generic.go:356] "Generic (PLEG): container finished" podID="19401fc8-ae1c-4249-9a75-d7aab8a904de" containerID="6ef4b3034329d7a1fb418b658c173917e1451999f5776138ef2c2922f9851265" exitCode=0
Sep 09 12:20:08.349315 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:08.349287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nvm9j" event={"ID":"19401fc8-ae1c-4249-9a75-d7aab8a904de","Type":"ContainerDied","Data":"6ef4b3034329d7a1fb418b658c173917e1451999f5776138ef2c2922f9851265"}
Sep 09 12:20:08.350392 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:08.349980    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:20:08.350392 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:08.350057    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:20:08.350392 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:08.350107    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:20:08.366558 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:08.366536    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:20:08.366685 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:08.366636    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:20:09.112387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:09.112196    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kxk9k"]
Sep 09 12:20:09.112566 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:09.112494    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:09.112625 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:09.112588    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:20:09.114788 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:09.114769    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdgrt"]
Sep 09 12:20:09.114873 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:09.114861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:09.114966 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:09.114944    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:20:09.118277 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:09.117650    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-thh9h"]
Sep 09 12:20:09.118277 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:09.117741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:09.118277 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:09.117840    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:20:09.353781 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:09.353688    2578 generic.go:356] "Generic (PLEG): container finished" podID="19401fc8-ae1c-4249-9a75-d7aab8a904de" containerID="038eef045b275012587cde8fb14e370e42264aae3ed265e6f59e4374d61a5170" exitCode=0
Sep 09 12:20:09.353781 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:09.353768    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nvm9j" event={"ID":"19401fc8-ae1c-4249-9a75-d7aab8a904de","Type":"ContainerDied","Data":"038eef045b275012587cde8fb14e370e42264aae3ed265e6f59e4374d61a5170"}
Sep 09 12:20:10.358805 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:10.358770    2578 generic.go:356] "Generic (PLEG): container finished" podID="19401fc8-ae1c-4249-9a75-d7aab8a904de" containerID="82d1c5df800017e154b9ef2af976eb22ccc1dc265462a37335d5b63b53656437" exitCode=0
Sep 09 12:20:10.359269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:10.358835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nvm9j" event={"ID":"19401fc8-ae1c-4249-9a75-d7aab8a904de","Type":"ContainerDied","Data":"82d1c5df800017e154b9ef2af976eb22ccc1dc265462a37335d5b63b53656437"}
Sep 09 12:20:11.180273 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:11.180083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:11.180443 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:11.180083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:11.180443 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:11.180363    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:20:11.180443 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:11.180417    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:20:11.180443 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:11.180120    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:11.180589 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:11.180506    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:20:13.180191 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:13.180100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:13.180628 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:13.180205    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:13.180628 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:13.180228    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kxk9k" podUID="004699ec-e8f0-4fb3-87b1-0d5a4ff3a610"
Sep 09 12:20:13.180628 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:13.180238    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:13.180628 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:13.180298    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdgrt" podUID="d09d81f3-b038-48e2-88c2-6ab36ddefce2"
Sep 09 12:20:13.180628 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:13.180405    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh9h" podUID="1a554455-a4e1-45a6-95d3-a129ec8da79b"
Sep 09 12:20:13.837775 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:13.837742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:13.838010 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:13.837895    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:20:13.838010 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:13.837951    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret podName:d09d81f3-b038-48e2-88c2-6ab36ddefce2 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:45.837936696 +0000 UTC m=+66.181534621 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret") pod "global-pull-secret-syncer-qdgrt" (UID: "d09d81f3-b038-48e2-88c2-6ab36ddefce2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:20:13.938646 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:13.938614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:13.938826 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:13.938695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlf4\" (UniqueName: \"kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4\") pod \"network-check-target-kxk9k\" (UID: \"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610\") " pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:13.938826 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:13.938782    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:20:13.938936 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:13.938834    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:20:13.938936 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:13.938854    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:20:13.938936 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:13.938857    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs podName:1a554455-a4e1-45a6-95d3-a129ec8da79b nodeName:}" failed. No retries permitted until 2026-09-09 12:20:45.938837206 +0000 UTC m=+66.282435138 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs") pod "network-metrics-daemon-thh9h" (UID: "1a554455-a4e1-45a6-95d3-a129ec8da79b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:20:13.938936 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:13.938867    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8jlf4 for pod openshift-network-diagnostics/network-check-target-kxk9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:20:13.938936 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:13.938918    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4 podName:004699ec-e8f0-4fb3-87b1-0d5a4ff3a610 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:45.938903192 +0000 UTC m=+66.282501121 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlf4" (UniqueName: "kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4") pod "network-check-target-kxk9k" (UID: "004699ec-e8f0-4fb3-87b1-0d5a4ff3a610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:20:14.511578 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.511537    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-152.ec2.internal" event="NodeReady"
Sep 09 12:20:14.512093 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.511707    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 12:20:14.547908 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.547876    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c4fb58b64-ghqtv"]
Sep 09 12:20:14.577682 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.577655    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"]
Sep 09 12:20:14.577864 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.577816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.580287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.580132    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 12:20:14.580482 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.580463    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bswb5\""
Sep 09 12:20:14.580572 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.580504    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 12:20:14.581696 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.580821    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 12:20:14.596489 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.596453    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxtvw"]
Sep 09 12:20:14.601885 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.601859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"
Sep 09 12:20:14.606085 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.605889    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 12:20:14.609284 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.608862    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 12:20:14.609284 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.609100    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 12:20:14.609284 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.609133    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 12:20:14.609284 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.608867    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:20:14.609550 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.609432    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7s6w9\""
Sep 09 12:20:14.621802 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.621775    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hzzgw"]
Sep 09 12:20:14.621912 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.621890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxtvw"
Sep 09 12:20:14.624208 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.624187    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:20:14.624334 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.624244    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-55phs\""
Sep 09 12:20:14.624334 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.624280    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 12:20:14.644545 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.644523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47v8t\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-kube-api-access-47v8t\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.644661 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.644564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-image-registry-private-configuration\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.644661 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.644592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-ca-trust-extracted\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.644661 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.644615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-certificates\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.644661 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.644649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-installation-pull-secrets\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.644862 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.644682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-trusted-ca\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.644862 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.644723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-bound-sa-token\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.644862 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.644752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.646247 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.646207    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"]
Sep 09 12:20:14.646350 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.646309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.648284 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.648257    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 12:20:14.648390 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.648258    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 12:20:14.648390 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.648258    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-69x42\""
Sep 09 12:20:14.648390 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.648320    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:20:14.648390 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.648356    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 12:20:14.652788 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.652766    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 12:20:14.668439 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.668420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:14.668439 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.668431    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6f8c9778b-t8sr5"]
Sep 09 12:20:14.670527 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.670510    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 12:20:14.671661 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.671639    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 12:20:14.671872 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.671853    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 12:20:14.671940 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.671864    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gcp85\""
Sep 09 12:20:14.672058 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.672044    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 12:20:14.683820 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.683800    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c4fb58b64-ghqtv"]
Sep 09 12:20:14.683925 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.683825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.683925 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.683843    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"]
Sep 09 12:20:14.685841 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.685820    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-56tvv\""
Sep 09 12:20:14.685967 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.685936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 12:20:14.686355 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.686339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 12:20:14.686442 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.686432    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 12:20:14.686548 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.686528    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 12:20:14.686640 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.686566    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 12:20:14.686640 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.686628    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 12:20:14.700032 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.700008    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f95hr"]
Sep 09 12:20:14.702444 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.702426    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"
Sep 09 12:20:14.705096 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.704865    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 12:20:14.705096 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.704951    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:20:14.705329 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.705305    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 12:20:14.705422 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.705318    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-92x58\""
Sep 09 12:20:14.705422 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.705359    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 12:20:14.718159 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.718133    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"]
Sep 09 12:20:14.718516 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.718500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.720268 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.720242    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 12:20:14.720598 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.720573    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 12:20:14.720994 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.720972    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hsrxf\""
Sep 09 12:20:14.733772 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.733754    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-tpw62"]
Sep 09 12:20:14.733897 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.733870    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:14.735678 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.735657    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 12:20:14.735834 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.735807    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 12:20:14.735936 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.735924    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4tssv\""
Sep 09 12:20:14.736088 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.736021    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:20:14.745196 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c00ec462-3123-4de8-b39f-297d8fa14545-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pgwwm\" (UID: \"c00ec462-3123-4de8-b39f-297d8fa14545\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"
Sep 09 12:20:14.745305 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47v8t\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-kube-api-access-47v8t\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.745305 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0548208b-e765-4614-b647-363d001d99c4-trusted-ca\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.745420 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-image-registry-private-configuration\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.745420 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-ca-trust-extracted\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.745420 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-certificates\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.745559 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-installation-pull-secrets\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.745559 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0548208b-e765-4614-b647-363d001d99c4-config\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.745559 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbvw\" (UniqueName: \"kubernetes.io/projected/0548208b-e765-4614-b647-363d001d99c4-kube-api-access-tnbvw\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.745559 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.745742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-trusted-ca\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.745742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0548208b-e765-4614-b647-363d001d99c4-serving-cert\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.745742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r4xl\" (UniqueName: \"kubernetes.io/projected/11252896-9746-4dbf-9ca6-01c7d854fab9-kube-api-access-4r4xl\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.745742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae1623ca-ad35-4f59-a2af-6e19fe017892-serving-cert\") pod \"service-ca-operator-859f94c4b-v5pn9\" (UID: \"ae1623ca-ad35-4f59-a2af-6e19fe017892\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"
Sep 09 12:20:14.745742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:14.745742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc96z\" (UniqueName: \"kubernetes.io/projected/ae1623ca-ad35-4f59-a2af-6e19fe017892-kube-api-access-xc96z\") pod \"service-ca-operator-859f94c4b-v5pn9\" (UID: \"ae1623ca-ad35-4f59-a2af-6e19fe017892\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"
Sep 09 12:20:14.745989 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qp54\" (UniqueName: \"kubernetes.io/projected/c00ec462-3123-4de8-b39f-297d8fa14545-kube-api-access-5qp54\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pgwwm\" (UID: \"c00ec462-3123-4de8-b39f-297d8fa14545\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"
Sep 09 12:20:14.745989 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4njn\" (UniqueName: \"kubernetes.io/projected/b538ac89-4973-401d-aedf-67b03ff11d0a-kube-api-access-z4njn\") pod \"volume-data-source-validator-695c895bd6-xxtvw\" (UID: \"b538ac89-4973-401d-aedf-67b03ff11d0a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxtvw"
Sep 09 12:20:14.745989 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-bound-sa-token\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.745989 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.745989 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.745933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c00ec462-3123-4de8-b39f-297d8fa14545-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pgwwm\" (UID: \"c00ec462-3123-4de8-b39f-297d8fa14545\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"
Sep 09 12:20:14.746236 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.746083    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 12:20:14.746236 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.746107    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4fb58b64-ghqtv: secret "image-registry-tls" not found
Sep 09 12:20:14.746236 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.746178    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls podName:4ebb9e23-a65c-4297-9e19-69bc45c4a54f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:15.246152665 +0000 UTC m=+35.589750591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls") pod "image-registry-7c4fb58b64-ghqtv" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f") : secret "image-registry-tls" not found
Sep 09 12:20:14.746236 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.746180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-ca-trust-extracted\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.746236 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.746231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.746448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.746257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae1623ca-ad35-4f59-a2af-6e19fe017892-config\") pod \"service-ca-operator-859f94c4b-v5pn9\" (UID: \"ae1623ca-ad35-4f59-a2af-6e19fe017892\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"
Sep 09 12:20:14.746448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.746279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0f9a503b-3e0d-4018-a745-aab35e92648f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:14.746448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.746301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-568hp\" (UniqueName: \"kubernetes.io/projected/0f9a503b-3e0d-4018-a745-aab35e92648f-kube-api-access-568hp\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:14.746448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.746328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-default-certificate\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.746448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.746350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-stats-auth\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.746448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.746351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-certificates\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.746704 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.746629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-trusted-ca\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.749872 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.749853    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxtvw"]
Sep 09 12:20:14.749978 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.749890    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-4ch9d"]
Sep 09 12:20:14.749978 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.749964    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.750087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.749982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-image-registry-private-configuration\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.750087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.750060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-installation-pull-secrets\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.752013 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.751992    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 12:20:14.752105 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.752018    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 12:20:14.752105 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.752001    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 12:20:14.752341 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.752324    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 12:20:14.752410 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.752343    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-kl4pw\""
Sep 09 12:20:14.756623 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.756604    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 12:20:14.758023 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.757992    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47v8t\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-kube-api-access-47v8t\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.762588 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.762537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-bound-sa-token\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:14.762887 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.762871    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-88dzs"]
Sep 09 12:20:14.762990 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.762976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:14.765128 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.765111    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 12:20:14.765243 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.765142    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 12:20:14.765243 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.765173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xkvm4\""
Sep 09 12:20:14.773337 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.773320    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ds22m"]
Sep 09 12:20:14.773460 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.773440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-88dzs"
Sep 09 12:20:14.775461 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.775442    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 12:20:14.775461 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.775461    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 12:20:14.775584 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.775468    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kls2h\""
Sep 09 12:20:14.775584 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.775496    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.781864    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"]
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.781893    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"]
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.781911    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hzzgw"]
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.781928    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"]
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.781943    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f95hr"]
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.781963    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-4ch9d"]
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.781967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-ds22m"
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.781982    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-88dzs"]
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.782001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-tpw62"]
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.782018    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f8c9778b-t8sr5"]
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.782035    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ds22m"]
Sep 09 12:20:14.782129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.782053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"]
Sep 09 12:20:14.783968 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.783947    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2wm7j\""
Sep 09 12:20:14.784066 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.783993    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 12:20:14.784066 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.783994    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 12:20:14.846976 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.846939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0548208b-e765-4614-b647-363d001d99c4-trusted-ca\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.846976 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.846985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.847238 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea6a22dd-7c89-4083-831d-3598aeb4bf58-serving-cert\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.847238 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847039    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-nginx-conf\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:14.847238 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:14.847238 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd502c7b-b814-4afd-8516-6d6e7fa25979-config-volume\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.847238 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:14.847238 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.847138    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:20:14.847238 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r4xl\" (UniqueName: \"kubernetes.io/projected/11252896-9746-4dbf-9ca6-01c7d854fab9-kube-api-access-4r4xl\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.847238 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae1623ca-ad35-4f59-a2af-6e19fe017892-serving-cert\") pod \"service-ca-operator-859f94c4b-v5pn9\" (UID: \"ae1623ca-ad35-4f59-a2af-6e19fe017892\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"
Sep 09 12:20:14.847238 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.847199    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:15.3471783 +0000 UTC m=+35.690776241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : secret "router-metrics-certs-default" not found
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qp54\" (UniqueName: \"kubernetes.io/projected/c00ec462-3123-4de8-b39f-297d8fa14545-kube-api-access-5qp54\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pgwwm\" (UID: \"c00ec462-3123-4de8-b39f-297d8fa14545\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4njn\" (UniqueName: \"kubernetes.io/projected/b538ac89-4973-401d-aedf-67b03ff11d0a-kube-api-access-z4njn\") pod \"volume-data-source-validator-695c895bd6-xxtvw\" (UID: \"b538ac89-4973-401d-aedf-67b03ff11d0a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxtvw"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps56d\" (UniqueName: \"kubernetes.io/projected/449412fc-55a9-44fd-81f1-6156d47e5381-kube-api-access-ps56d\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6a22dd-7c89-4083-831d-3598aeb4bf58-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq7z9\" (UniqueName: \"kubernetes.io/projected/ea6a22dd-7c89-4083-831d-3598aeb4bf58-kube-api-access-xq7z9\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc96z\" (UniqueName: \"kubernetes.io/projected/ae1623ca-ad35-4f59-a2af-6e19fe017892-kube-api-access-xc96z\") pod \"service-ca-operator-859f94c4b-v5pn9\" (UID: \"ae1623ca-ad35-4f59-a2af-6e19fe017892\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0f9a503b-3e0d-4018-a745-aab35e92648f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-568hp\" (UniqueName: \"kubernetes.io/projected/0f9a503b-3e0d-4018-a745-aab35e92648f-kube-api-access-568hp\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxntb\" (UniqueName: \"kubernetes.io/projected/d7e96ad5-f31e-45dd-b44e-e49453638364-kube-api-access-sxntb\") pod \"ingress-canary-88dzs\" (UID: \"d7e96ad5-f31e-45dd-b44e-e49453638364\") " pod="openshift-ingress-canary/ingress-canary-88dzs"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h846j\" (UniqueName: \"kubernetes.io/projected/fd502c7b-b814-4afd-8516-6d6e7fa25979-kube-api-access-h846j\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.847564    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:15.347545685 +0000 UTC m=+35.691143627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : configmap references non-existent config key: service-ca.crt
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea6a22dd-7c89-4083-831d-3598aeb4bf58-tmp\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.847771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6a22dd-7c89-4083-831d-3598aeb4bf58-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae1623ca-ad35-4f59-a2af-6e19fe017892-config\") pod \"service-ca-operator-859f94c4b-v5pn9\" (UID: \"ae1623ca-ad35-4f59-a2af-6e19fe017892\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-default-certificate\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0548208b-e765-4614-b647-363d001d99c4-config\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd502c7b-b814-4afd-8516-6d6e7fa25979-tmp-dir\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbvw\" (UniqueName: \"kubernetes.io/projected/0548208b-e765-4614-b647-363d001d99c4-kube-api-access-tnbvw\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0548208b-e765-4614-b647-363d001d99c4-serving-cert\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea6a22dd-7c89-4083-831d-3598aeb4bf58-snapshots\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0548208b-e765-4614-b647-363d001d99c4-trusted-ca\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.847980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7e96ad5-f31e-45dd-b44e-e49453638364-cert\") pod \"ingress-canary-88dzs\" (UID: \"d7e96ad5-f31e-45dd-b44e-e49453638364\") " pod="openshift-ingress-canary/ingress-canary-88dzs"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.848023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c00ec462-3123-4de8-b39f-297d8fa14545-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pgwwm\" (UID: \"c00ec462-3123-4de8-b39f-297d8fa14545\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.848049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.848055    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.848080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk98v\" (UniqueName: \"kubernetes.io/projected/a4870d82-0767-4124-944b-87632c97f753-kube-api-access-tk98v\") pod \"network-check-source-685456655f-ds22m\" (UID: \"a4870d82-0767-4124-944b-87632c97f753\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ds22m"
Sep 09 12:20:14.848440 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.848117    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls podName:0f9a503b-3e0d-4018-a745-aab35e92648f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:15.348101629 +0000 UTC m=+35.691699578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-rc7bk" (UID: "0f9a503b-3e0d-4018-a745-aab35e92648f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:14.849101 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.848136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c00ec462-3123-4de8-b39f-297d8fa14545-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pgwwm\" (UID: \"c00ec462-3123-4de8-b39f-297d8fa14545\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"
Sep 09 12:20:14.849101 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.848174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-stats-auth\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.849101 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.848578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0f9a503b-3e0d-4018-a745-aab35e92648f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:14.849484 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.849456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c00ec462-3123-4de8-b39f-297d8fa14545-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pgwwm\" (UID: \"c00ec462-3123-4de8-b39f-297d8fa14545\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"
Sep 09 12:20:14.850720 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.849804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0548208b-e765-4614-b647-363d001d99c4-config\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.850720 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.848209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae1623ca-ad35-4f59-a2af-6e19fe017892-config\") pod \"service-ca-operator-859f94c4b-v5pn9\" (UID: \"ae1623ca-ad35-4f59-a2af-6e19fe017892\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"
Sep 09 12:20:14.850720 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.850033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae1623ca-ad35-4f59-a2af-6e19fe017892-serving-cert\") pod \"service-ca-operator-859f94c4b-v5pn9\" (UID: \"ae1623ca-ad35-4f59-a2af-6e19fe017892\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"
Sep 09 12:20:14.855821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.855795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0548208b-e765-4614-b647-363d001d99c4-serving-cert\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.855934 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.855845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-default-certificate\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.856069 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.856044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c00ec462-3123-4de8-b39f-297d8fa14545-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pgwwm\" (UID: \"c00ec462-3123-4de8-b39f-297d8fa14545\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"
Sep 09 12:20:14.859532 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.859507    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc96z\" (UniqueName: \"kubernetes.io/projected/ae1623ca-ad35-4f59-a2af-6e19fe017892-kube-api-access-xc96z\") pod \"service-ca-operator-859f94c4b-v5pn9\" (UID: \"ae1623ca-ad35-4f59-a2af-6e19fe017892\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"
Sep 09 12:20:14.859532 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.859521    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qp54\" (UniqueName: \"kubernetes.io/projected/c00ec462-3123-4de8-b39f-297d8fa14545-kube-api-access-5qp54\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pgwwm\" (UID: \"c00ec462-3123-4de8-b39f-297d8fa14545\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"
Sep 09 12:20:14.859684 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.859605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-stats-auth\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.859803 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.859784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-568hp\" (UniqueName: \"kubernetes.io/projected/0f9a503b-3e0d-4018-a745-aab35e92648f-kube-api-access-568hp\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:14.860100 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.860079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4njn\" (UniqueName: \"kubernetes.io/projected/b538ac89-4973-401d-aedf-67b03ff11d0a-kube-api-access-z4njn\") pod \"volume-data-source-validator-695c895bd6-xxtvw\" (UID: \"b538ac89-4973-401d-aedf-67b03ff11d0a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxtvw"
Sep 09 12:20:14.861113 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.861085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r4xl\" (UniqueName: \"kubernetes.io/projected/11252896-9746-4dbf-9ca6-01c7d854fab9-kube-api-access-4r4xl\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:14.861765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.861741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbvw\" (UniqueName: \"kubernetes.io/projected/0548208b-e765-4614-b647-363d001d99c4-kube-api-access-tnbvw\") pod \"console-operator-85b454874b-hzzgw\" (UID: \"0548208b-e765-4614-b647-363d001d99c4\") " pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.919571 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.919536    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"
Sep 09 12:20:14.931665 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.931634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxtvw"
Sep 09 12:20:14.949546 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd502c7b-b814-4afd-8516-6d6e7fa25979-tmp-dir\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.949651 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea6a22dd-7c89-4083-831d-3598aeb4bf58-snapshots\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.949651 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7e96ad5-f31e-45dd-b44e-e49453638364-cert\") pod \"ingress-canary-88dzs\" (UID: \"d7e96ad5-f31e-45dd-b44e-e49453638364\") " pod="openshift-ingress-canary/ingress-canary-88dzs"
Sep 09 12:20:14.949651 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949615    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.949651 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk98v\" (UniqueName: \"kubernetes.io/projected/a4870d82-0767-4124-944b-87632c97f753-kube-api-access-tk98v\") pod \"network-check-source-685456655f-ds22m\" (UID: \"a4870d82-0767-4124-944b-87632c97f753\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ds22m"
Sep 09 12:20:14.949844 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea6a22dd-7c89-4083-831d-3598aeb4bf58-serving-cert\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.949844 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-nginx-conf\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:14.949844 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:14.949844 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.949762    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:20:14.950018 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd502c7b-b814-4afd-8516-6d6e7fa25979-tmp-dir\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.950018 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.949845    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls podName:fd502c7b-b814-4afd-8516-6d6e7fa25979 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:15.449823881 +0000 UTC m=+35.793421822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls") pod "dns-default-f95hr" (UID: "fd502c7b-b814-4afd-8516-6d6e7fa25979") : secret "dns-default-metrics-tls" not found
Sep 09 12:20:14.950018 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd502c7b-b814-4afd-8516-6d6e7fa25979-config-volume\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.950018 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:14.950018 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps56d\" (UniqueName: \"kubernetes.io/projected/449412fc-55a9-44fd-81f1-6156d47e5381-kube-api-access-ps56d\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:14.950018 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.949990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6a22dd-7c89-4083-831d-3598aeb4bf58-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.950018 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.950016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq7z9\" (UniqueName: \"kubernetes.io/projected/ea6a22dd-7c89-4083-831d-3598aeb4bf58-kube-api-access-xq7z9\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.950359 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.950078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxntb\" (UniqueName: \"kubernetes.io/projected/d7e96ad5-f31e-45dd-b44e-e49453638364-kube-api-access-sxntb\") pod \"ingress-canary-88dzs\" (UID: \"d7e96ad5-f31e-45dd-b44e-e49453638364\") " pod="openshift-ingress-canary/ingress-canary-88dzs"
Sep 09 12:20:14.950359 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.950109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h846j\" (UniqueName: \"kubernetes.io/projected/fd502c7b-b814-4afd-8516-6d6e7fa25979-kube-api-access-h846j\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.950359 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.950131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea6a22dd-7c89-4083-831d-3598aeb4bf58-tmp\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.950359 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.950151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6a22dd-7c89-4083-831d-3598aeb4bf58-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.950359 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.950164    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 12:20:14.950359 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.950236    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls podName:449412fc-55a9-44fd-81f1-6156d47e5381 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:15.450199547 +0000 UTC m=+35.793797485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j444k" (UID: "449412fc-55a9-44fd-81f1-6156d47e5381") : secret "samples-operator-tls" not found
Sep 09 12:20:14.950359 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.950294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd502c7b-b814-4afd-8516-6d6e7fa25979-config-volume\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.950946 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.950675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea6a22dd-7c89-4083-831d-3598aeb4bf58-snapshots\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.950946 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.950740    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 12:20:14.950946 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:14.950781    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert podName:6cb90a7b-a57f-4dc3-9231-c74b11e6173e nodeName:}" failed. No retries permitted until 2026-09-09 12:20:15.450771354 +0000 UTC m=+35.794369280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4ch9d" (UID: "6cb90a7b-a57f-4dc3-9231-c74b11e6173e") : secret "networking-console-plugin-cert" not found
Sep 09 12:20:14.950946 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.950849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-nginx-conf\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:14.950946 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.950932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea6a22dd-7c89-4083-831d-3598aeb4bf58-tmp\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.951259 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.951164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6a22dd-7c89-4083-831d-3598aeb4bf58-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.951312 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.951268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6a22dd-7c89-4083-831d-3598aeb4bf58-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.952429 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.952411    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea6a22dd-7c89-4083-831d-3598aeb4bf58-serving-cert\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.952700 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.952678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7e96ad5-f31e-45dd-b44e-e49453638364-cert\") pod \"ingress-canary-88dzs\" (UID: \"d7e96ad5-f31e-45dd-b44e-e49453638364\") " pod="openshift-ingress-canary/ingress-canary-88dzs"
Sep 09 12:20:14.955917 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.955894    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:14.959281 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.959258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps56d\" (UniqueName: \"kubernetes.io/projected/449412fc-55a9-44fd-81f1-6156d47e5381-kube-api-access-ps56d\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:14.959387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.959359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h846j\" (UniqueName: \"kubernetes.io/projected/fd502c7b-b814-4afd-8516-6d6e7fa25979-kube-api-access-h846j\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:14.960419 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.960396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq7z9\" (UniqueName: \"kubernetes.io/projected/ea6a22dd-7c89-4083-831d-3598aeb4bf58-kube-api-access-xq7z9\") pod \"insights-operator-76bcd6ccf5-tpw62\" (UID: \"ea6a22dd-7c89-4083-831d-3598aeb4bf58\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:14.960496 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.960418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk98v\" (UniqueName: \"kubernetes.io/projected/a4870d82-0767-4124-944b-87632c97f753-kube-api-access-tk98v\") pod \"network-check-source-685456655f-ds22m\" (UID: \"a4870d82-0767-4124-944b-87632c97f753\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ds22m"
Sep 09 12:20:14.960564 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:14.960547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxntb\" (UniqueName: \"kubernetes.io/projected/d7e96ad5-f31e-45dd-b44e-e49453638364-kube-api-access-sxntb\") pod \"ingress-canary-88dzs\" (UID: \"d7e96ad5-f31e-45dd-b44e-e49453638364\") " pod="openshift-ingress-canary/ingress-canary-88dzs"
Sep 09 12:20:15.012814 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.012750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"
Sep 09 12:20:15.077442 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.077408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62"
Sep 09 12:20:15.092695 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.092664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-88dzs"
Sep 09 12:20:15.098651 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.098592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-ds22m"
Sep 09 12:20:15.180232 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.180189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:15.180398 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.180273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:15.180398 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.180289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:15.182969 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.182946    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 12:20:15.183119 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.182999    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 12:20:15.183119 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.183056    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lblnd\""
Sep 09 12:20:15.183401 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.183379    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wjmxh\""
Sep 09 12:20:15.253160 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.253124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:15.253349 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.253328    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 12:20:15.253349 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.253347    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4fb58b64-ghqtv: secret "image-registry-tls" not found
Sep 09 12:20:15.253463 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.253425    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls podName:4ebb9e23-a65c-4297-9e19-69bc45c4a54f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:16.253404092 +0000 UTC m=+36.597002021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls") pod "image-registry-7c4fb58b64-ghqtv" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f") : secret "image-registry-tls" not found
Sep 09 12:20:15.354078 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.353967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:15.354269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.354103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:15.354269 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.354142    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:15.354269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.354170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:15.354269 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.354235    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls podName:0f9a503b-3e0d-4018-a745-aab35e92648f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:16.354198933 +0000 UTC m=+36.697796872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-rc7bk" (UID: "0f9a503b-3e0d-4018-a745-aab35e92648f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:15.354509 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.354279    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:20:15.354509 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.354314    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:16.35429552 +0000 UTC m=+36.697893472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : configmap references non-existent config key: service-ca.crt
Sep 09 12:20:15.354509 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.354335    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:16.354325592 +0000 UTC m=+36.697923519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : secret "router-metrics-certs-default" not found
Sep 09 12:20:15.455328 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.455299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:15.455521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.455335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:15.455521 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.455462    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 12:20:15.455521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.455489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:15.455678 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.455526    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls podName:449412fc-55a9-44fd-81f1-6156d47e5381 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:16.455508263 +0000 UTC m=+36.799106206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j444k" (UID: "449412fc-55a9-44fd-81f1-6156d47e5381") : secret "samples-operator-tls" not found
Sep 09 12:20:15.455678 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.455539    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 12:20:15.455678 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.455576    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:20:15.455678 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.455598    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert podName:6cb90a7b-a57f-4dc3-9231-c74b11e6173e nodeName:}" failed. No retries permitted until 2026-09-09 12:20:16.455580582 +0000 UTC m=+36.799178522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4ch9d" (UID: "6cb90a7b-a57f-4dc3-9231-c74b11e6173e") : secret "networking-console-plugin-cert" not found
Sep 09 12:20:15.455678 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:15.455614    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls podName:fd502c7b-b814-4afd-8516-6d6e7fa25979 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:16.455604079 +0000 UTC m=+36.799202008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls") pod "dns-default-f95hr" (UID: "fd502c7b-b814-4afd-8516-6d6e7fa25979") : secret "dns-default-metrics-tls" not found
Sep 09 12:20:15.754161 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.753924    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxtvw"]
Sep 09 12:20:15.756319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.756259    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hzzgw"]
Sep 09 12:20:15.758279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.757597    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ds22m"]
Sep 09 12:20:15.758612 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.758595    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9"]
Sep 09 12:20:15.764469 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:15.764443    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4870d82_0767_4124_944b_87632c97f753.slice/crio-1e2a8ef605e69268b11184f036ed7b1d2829662bc0e09e842fd2b18eadbb4329 WatchSource:0}: Error finding container 1e2a8ef605e69268b11184f036ed7b1d2829662bc0e09e842fd2b18eadbb4329: Status 404 returned error can't find the container with id 1e2a8ef605e69268b11184f036ed7b1d2829662bc0e09e842fd2b18eadbb4329
Sep 09 12:20:15.765260 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:15.765133    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1623ca_ad35_4f59_a2af_6e19fe017892.slice/crio-2b8bc124a74928a5ac12af403f80ae8ec035145df9f17de06e3d7502cfa12b5f WatchSource:0}: Error finding container 2b8bc124a74928a5ac12af403f80ae8ec035145df9f17de06e3d7502cfa12b5f: Status 404 returned error can't find the container with id 2b8bc124a74928a5ac12af403f80ae8ec035145df9f17de06e3d7502cfa12b5f
Sep 09 12:20:15.770799 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.770778    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-88dzs"]
Sep 09 12:20:15.774854 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.774831    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-tpw62"]
Sep 09 12:20:15.776805 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:15.776737    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e96ad5_f31e_45dd_b44e_e49453638364.slice/crio-2bbe182406e17c94b47e57c95857aafe2109b86be8c84460bd26d0a3cddcedf0 WatchSource:0}: Error finding container 2bbe182406e17c94b47e57c95857aafe2109b86be8c84460bd26d0a3cddcedf0: Status 404 returned error can't find the container with id 2bbe182406e17c94b47e57c95857aafe2109b86be8c84460bd26d0a3cddcedf0
Sep 09 12:20:15.777339 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:15.777001    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6a22dd_7c89_4083_831d_3598aeb4bf58.slice/crio-418141adac7ef7fa9dd1bb6919537c3cffee16c854201821f9efa5fb4c35aff4 WatchSource:0}: Error finding container 418141adac7ef7fa9dd1bb6919537c3cffee16c854201821f9efa5fb4c35aff4: Status 404 returned error can't find the container with id 418141adac7ef7fa9dd1bb6919537c3cffee16c854201821f9efa5fb4c35aff4
Sep 09 12:20:15.786650 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:15.786615    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm"]
Sep 09 12:20:15.790551 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:15.790524    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00ec462_3123_4de8_b39f_297d8fa14545.slice/crio-96b6399e6282df9d8251aba5498d404b70c4e82a9fb774e4c3f9e303137a9223 WatchSource:0}: Error finding container 96b6399e6282df9d8251aba5498d404b70c4e82a9fb774e4c3f9e303137a9223: Status 404 returned error can't find the container with id 96b6399e6282df9d8251aba5498d404b70c4e82a9fb774e4c3f9e303137a9223
Sep 09 12:20:16.263640 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.263568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:16.263813 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.263739    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 12:20:16.263813 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.263759    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4fb58b64-ghqtv: secret "image-registry-tls" not found
Sep 09 12:20:16.263925 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.263819    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls podName:4ebb9e23-a65c-4297-9e19-69bc45c4a54f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:18.263798844 +0000 UTC m=+38.607396789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls") pod "image-registry-7c4fb58b64-ghqtv" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f") : secret "image-registry-tls" not found
Sep 09 12:20:16.364745 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.364713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:16.364907 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.364792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:16.364907 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.364858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:16.364907 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.364878    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:20:16.365009 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.364944    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:18.3649289 +0000 UTC m=+38.708526830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : secret "router-metrics-certs-default" not found
Sep 09 12:20:16.365009 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.364966    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:18.364949725 +0000 UTC m=+38.708547670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : configmap references non-existent config key: service-ca.crt
Sep 09 12:20:16.365009 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.364986    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:16.365128 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.365022    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls podName:0f9a503b-3e0d-4018-a745-aab35e92648f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:18.365014484 +0000 UTC m=+38.708612410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-rc7bk" (UID: "0f9a503b-3e0d-4018-a745-aab35e92648f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:16.375508 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.375444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm" event={"ID":"c00ec462-3123-4de8-b39f-297d8fa14545","Type":"ContainerStarted","Data":"96b6399e6282df9d8251aba5498d404b70c4e82a9fb774e4c3f9e303137a9223"}
Sep 09 12:20:16.376887 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.376834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hzzgw" event={"ID":"0548208b-e765-4614-b647-363d001d99c4","Type":"ContainerStarted","Data":"7673ec124e97476f9e07f68dfb251b67a36a5fbbc004317bf65eec32adb1a4a3"}
Sep 09 12:20:16.380372 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.380348    2578 generic.go:356] "Generic (PLEG): container finished" podID="19401fc8-ae1c-4249-9a75-d7aab8a904de" containerID="98257e554cfc15883528ecf168531f6008a025b65fea782d24fd8c0c4d73f4cd" exitCode=0
Sep 09 12:20:16.380511 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.380441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nvm9j" event={"ID":"19401fc8-ae1c-4249-9a75-d7aab8a904de","Type":"ContainerDied","Data":"98257e554cfc15883528ecf168531f6008a025b65fea782d24fd8c0c4d73f4cd"}
Sep 09 12:20:16.382041 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.382022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxtvw" event={"ID":"b538ac89-4973-401d-aedf-67b03ff11d0a","Type":"ContainerStarted","Data":"91899aa4c72e32ff2276cb555050ca65392b66cbe2dffd75de7536894b3681ae"}
Sep 09 12:20:16.383383 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.383359    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9" event={"ID":"ae1623ca-ad35-4f59-a2af-6e19fe017892","Type":"ContainerStarted","Data":"2b8bc124a74928a5ac12af403f80ae8ec035145df9f17de06e3d7502cfa12b5f"}
Sep 09 12:20:16.384719 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.384696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62" event={"ID":"ea6a22dd-7c89-4083-831d-3598aeb4bf58","Type":"ContainerStarted","Data":"418141adac7ef7fa9dd1bb6919537c3cffee16c854201821f9efa5fb4c35aff4"}
Sep 09 12:20:16.386476 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.386431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-ds22m" event={"ID":"a4870d82-0767-4124-944b-87632c97f753","Type":"ContainerStarted","Data":"1e2a8ef605e69268b11184f036ed7b1d2829662bc0e09e842fd2b18eadbb4329"}
Sep 09 12:20:16.387835 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.387812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-88dzs" event={"ID":"d7e96ad5-f31e-45dd-b44e-e49453638364","Type":"ContainerStarted","Data":"2bbe182406e17c94b47e57c95857aafe2109b86be8c84460bd26d0a3cddcedf0"}
Sep 09 12:20:16.466758 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.465488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:16.466758 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.465702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:16.466758 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:16.465787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:16.466758 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.465927    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 12:20:16.466758 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.465988    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls podName:449412fc-55a9-44fd-81f1-6156d47e5381 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:18.465970198 +0000 UTC m=+38.809568124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j444k" (UID: "449412fc-55a9-44fd-81f1-6156d47e5381") : secret "samples-operator-tls" not found
Sep 09 12:20:16.466758 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.466386    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:20:16.466758 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.466437    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls podName:fd502c7b-b814-4afd-8516-6d6e7fa25979 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:18.46642145 +0000 UTC m=+38.810019376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls") pod "dns-default-f95hr" (UID: "fd502c7b-b814-4afd-8516-6d6e7fa25979") : secret "dns-default-metrics-tls" not found
Sep 09 12:20:16.466758 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.466715    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 12:20:16.466758 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:16.466753    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert podName:6cb90a7b-a57f-4dc3-9231-c74b11e6173e nodeName:}" failed. No retries permitted until 2026-09-09 12:20:18.466739811 +0000 UTC m=+38.810337737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4ch9d" (UID: "6cb90a7b-a57f-4dc3-9231-c74b11e6173e") : secret "networking-console-plugin-cert" not found
Sep 09 12:20:17.411204 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:17.411165    2578 generic.go:356] "Generic (PLEG): container finished" podID="19401fc8-ae1c-4249-9a75-d7aab8a904de" containerID="cd00a695307d74c4d15d3c917c9666f497096376aaf658cc8f69c330ce744091" exitCode=0
Sep 09 12:20:17.411753 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:17.411251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nvm9j" event={"ID":"19401fc8-ae1c-4249-9a75-d7aab8a904de","Type":"ContainerDied","Data":"cd00a695307d74c4d15d3c917c9666f497096376aaf658cc8f69c330ce744091"}
Sep 09 12:20:18.288425 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:18.287854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:18.288425 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.287999    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 12:20:18.288425 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.288014    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4fb58b64-ghqtv: secret "image-registry-tls" not found
Sep 09 12:20:18.288425 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.288069    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls podName:4ebb9e23-a65c-4297-9e19-69bc45c4a54f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:22.288051059 +0000 UTC m=+42.631648987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls") pod "image-registry-7c4fb58b64-ghqtv" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f") : secret "image-registry-tls" not found
Sep 09 12:20:18.389315 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:18.389280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:18.389487 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:18.389378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:18.389487 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:18.389454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:18.389771 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.389635    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:18.389771 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.389703    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls podName:0f9a503b-3e0d-4018-a745-aab35e92648f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:22.389683859 +0000 UTC m=+42.733281788 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-rc7bk" (UID: "0f9a503b-3e0d-4018-a745-aab35e92648f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:18.389771 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.389704    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:20:18.389771 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.389737    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:22.389715643 +0000 UTC m=+42.733313572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : configmap references non-existent config key: service-ca.crt
Sep 09 12:20:18.389771 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.389763    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:22.389752637 +0000 UTC m=+42.733350564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : secret "router-metrics-certs-default" not found
Sep 09 12:20:18.490941 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:18.490899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:18.491405 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:18.490954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:18.491405 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:18.491051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:18.491405 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.491079    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 12:20:18.491405 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.491143    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:20:18.491405 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.491161    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls podName:449412fc-55a9-44fd-81f1-6156d47e5381 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:22.491139495 +0000 UTC m=+42.834737423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j444k" (UID: "449412fc-55a9-44fd-81f1-6156d47e5381") : secret "samples-operator-tls" not found
Sep 09 12:20:18.491405 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.491180    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls podName:fd502c7b-b814-4afd-8516-6d6e7fa25979 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:22.49116911 +0000 UTC m=+42.834767036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls") pod "dns-default-f95hr" (UID: "fd502c7b-b814-4afd-8516-6d6e7fa25979") : secret "dns-default-metrics-tls" not found
Sep 09 12:20:18.491405 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.491236    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 12:20:18.491405 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:18.491270    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert podName:6cb90a7b-a57f-4dc3-9231-c74b11e6173e nodeName:}" failed. No retries permitted until 2026-09-09 12:20:22.491259079 +0000 UTC m=+42.834857005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4ch9d" (UID: "6cb90a7b-a57f-4dc3-9231-c74b11e6173e") : secret "networking-console-plugin-cert" not found
Sep 09 12:20:22.329540 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:22.329497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:22.329986 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.329674    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 12:20:22.329986 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.329700    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4fb58b64-ghqtv: secret "image-registry-tls" not found
Sep 09 12:20:22.329986 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.329779    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls podName:4ebb9e23-a65c-4297-9e19-69bc45c4a54f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:30.329756689 +0000 UTC m=+50.673354618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls") pod "image-registry-7c4fb58b64-ghqtv" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f") : secret "image-registry-tls" not found
Sep 09 12:20:22.430280 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:22.430245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:22.430472 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:22.430338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:22.430472 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.430387    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:20:22.430472 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:22.430416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:22.430619 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.430484    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:30.430464707 +0000 UTC m=+50.774062635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : secret "router-metrics-certs-default" not found
Sep 09 12:20:22.430619 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.430526    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:22.430619 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.430582    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls podName:0f9a503b-3e0d-4018-a745-aab35e92648f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:30.430564356 +0000 UTC m=+50.774162302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-rc7bk" (UID: "0f9a503b-3e0d-4018-a745-aab35e92648f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:22.430762 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.430641    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:30.430629623 +0000 UTC m=+50.774227563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : configmap references non-existent config key: service-ca.crt
Sep 09 12:20:22.531948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:22.531906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:22.532122 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:22.532000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:22.532122 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:22.532032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:22.532122 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.532065    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:20:22.532326 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.532145    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls podName:fd502c7b-b814-4afd-8516-6d6e7fa25979 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:30.532122992 +0000 UTC m=+50.875720935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls") pod "dns-default-f95hr" (UID: "fd502c7b-b814-4afd-8516-6d6e7fa25979") : secret "dns-default-metrics-tls" not found
Sep 09 12:20:22.532326 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.532168    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 12:20:22.532326 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.532185    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 12:20:22.532326 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.532243    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls podName:449412fc-55a9-44fd-81f1-6156d47e5381 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:30.532229335 +0000 UTC m=+50.875827287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j444k" (UID: "449412fc-55a9-44fd-81f1-6156d47e5381") : secret "samples-operator-tls" not found
Sep 09 12:20:22.532326 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:22.532258    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert podName:6cb90a7b-a57f-4dc3-9231-c74b11e6173e nodeName:}" failed. No retries permitted until 2026-09-09 12:20:30.532251793 +0000 UTC m=+50.875849719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4ch9d" (UID: "6cb90a7b-a57f-4dc3-9231-c74b11e6173e") : secret "networking-console-plugin-cert" not found
Sep 09 12:20:24.440926 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.440293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nvm9j" event={"ID":"19401fc8-ae1c-4249-9a75-d7aab8a904de","Type":"ContainerStarted","Data":"f30559a8b7fd456e21a838fb02c08d797bfb264e221c9f165fad57e322eaf7d0"}
Sep 09 12:20:24.447001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.446971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxtvw" event={"ID":"b538ac89-4973-401d-aedf-67b03ff11d0a","Type":"ContainerStarted","Data":"a39ad815ec1d0c702b58203a988bdeabf4d87fe4ac54f0517570a0a4d578647a"}
Sep 09 12:20:24.449235 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.449161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9" event={"ID":"ae1623ca-ad35-4f59-a2af-6e19fe017892","Type":"ContainerStarted","Data":"da0faf4bb13e3c58ddf750acb7a283682b9e1fb443d938f0f9ba5bcca15984dc"}
Sep 09 12:20:24.450763 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.450740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62" event={"ID":"ea6a22dd-7c89-4083-831d-3598aeb4bf58","Type":"ContainerStarted","Data":"6d909fb8fd8119c44ed717fa2fc9abb7af6eed6b5f9447dd0c83a5bd328b68a9"}
Sep 09 12:20:24.452373 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.452346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-ds22m" event={"ID":"a4870d82-0767-4124-944b-87632c97f753","Type":"ContainerStarted","Data":"04adc50df1a44fbaa7f48aa389b70a27e80a080dafd1441684a8561a19b38966"}
Sep 09 12:20:24.454132 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.454111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-88dzs" event={"ID":"d7e96ad5-f31e-45dd-b44e-e49453638364","Type":"ContainerStarted","Data":"25186ca8136208e59cae0ccc762e90da5ccad1c768563ab03662ca93d241b661"}
Sep 09 12:20:24.459252 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.459230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm" event={"ID":"c00ec462-3123-4de8-b39f-297d8fa14545","Type":"ContainerStarted","Data":"acb3a34cab60c452f937eb3e8607344ba8570488066a0b786c6deb020ff8aaa4"}
Sep 09 12:20:24.461656 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.461640    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/0.log"
Sep 09 12:20:24.461755 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.461667    2578 generic.go:356] "Generic (PLEG): container finished" podID="0548208b-e765-4614-b647-363d001d99c4" containerID="48bffbc0df77c5306e7cacf519b28c4e41895ec9ac9008460b42f35af00aeb41" exitCode=255
Sep 09 12:20:24.461755 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.461688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hzzgw" event={"ID":"0548208b-e765-4614-b647-363d001d99c4","Type":"ContainerDied","Data":"48bffbc0df77c5306e7cacf519b28c4e41895ec9ac9008460b42f35af00aeb41"}
Sep 09 12:20:24.461900 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.461888    2578 scope.go:117] "RemoveContainer" containerID="48bffbc0df77c5306e7cacf519b28c4e41895ec9ac9008460b42f35af00aeb41"
Sep 09 12:20:24.465794 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.465752    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nvm9j" podStartSLOduration=13.782870182 podStartE2EDuration="44.465736739s" totalImagesPullingTime="30.682866557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:24.46440997 +0000 UTC m=+44.808007919" watchObservedRunningTime="2026-09-09 12:20:24.465736739 +0000 UTC m=+44.809334690"
Sep 09 12:20:24.500788 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.500739    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxtvw" podStartSLOduration=14.614762449 podStartE2EDuration="22.50071988s" totalImagesPullingTime="7.885957431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:24.48183865 +0000 UTC m=+44.825436597" watchObservedRunningTime="2026-09-09 12:20:24.50071988 +0000 UTC m=+44.844317830"
Sep 09 12:20:24.529230 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.527532    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-ds22m" podStartSLOduration=14.619145397 podStartE2EDuration="22.527512544s" totalImagesPullingTime="7.908367147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:24.526972723 +0000 UTC m=+44.870570668" watchObservedRunningTime="2026-09-09 12:20:24.527512544 +0000 UTC m=+44.871110494"
Sep 09 12:20:24.529230 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.528700    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-88dzs" podStartSLOduration=2.6444687 podStartE2EDuration="10.528686473s" totalImagesPullingTime="7.884217773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:24.50112224 +0000 UTC m=+44.844720172" watchObservedRunningTime="2026-09-09 12:20:24.528686473 +0000 UTC m=+44.872284422"
Sep 09 12:20:24.567880 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.567828    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm" podStartSLOduration=14.697241034 podStartE2EDuration="22.567808087s" totalImagesPullingTime="7.870567053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:24.565792336 +0000 UTC m=+44.909390299" watchObservedRunningTime="2026-09-09 12:20:24.567808087 +0000 UTC m=+44.911406038"
Sep 09 12:20:24.610269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.604649    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62" podStartSLOduration=14.721384361 podStartE2EDuration="22.604628944s" totalImagesPullingTime="7.883244583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:24.584975185 +0000 UTC m=+44.928573137" watchObservedRunningTime="2026-09-09 12:20:24.604628944 +0000 UTC m=+44.948226893"
Sep 09 12:20:24.610269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.606104    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9" podStartSLOduration=14.714210642 podStartE2EDuration="22.606091413s" totalImagesPullingTime="7.891880771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:24.604151459 +0000 UTC m=+44.947749405" watchObservedRunningTime="2026-09-09 12:20:24.606091413 +0000 UTC m=+44.949689362"
Sep 09 12:20:24.956100 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.956058    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:24.956391 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:24.956127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:25.161079 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.161045    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vl2k6"]
Sep 09 12:20:25.182375 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.182340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vl2k6"]
Sep 09 12:20:25.182535 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.182449    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.186182 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.186155    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 12:20:25.186462 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.186444    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 12:20:25.186549 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.186507    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tndcj\""
Sep 09 12:20:25.360343 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.360239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxzrm\" (UniqueName: \"kubernetes.io/projected/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-api-access-zxzrm\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.360343 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.360336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.360564 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.360395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c86c595-fb8e-45e3-b6c0-cb9871af938d-data-volume\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.360564 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.360432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.360564 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.360460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c86c595-fb8e-45e3-b6c0-cb9871af938d-crio-socket\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.461312 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.461264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.461780 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.461382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c86c595-fb8e-45e3-b6c0-cb9871af938d-data-volume\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.461780 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:25.461400    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:20:25.461780 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.461414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.461780 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.461459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c86c595-fb8e-45e3-b6c0-cb9871af938d-crio-socket\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.461780 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:25.461525    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:20:25.461780 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.461539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c86c595-fb8e-45e3-b6c0-cb9871af938d-crio-socket\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.461780 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:25.461577    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm podName:3c86c595-fb8e-45e3-b6c0-cb9871af938d nodeName:}" failed. No retries permitted until 2026-09-09 12:20:25.961557474 +0000 UTC m=+46.305155402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vl2k6" (UID: "3c86c595-fb8e-45e3-b6c0-cb9871af938d") : configmap "kube-rbac-proxy" not found
Sep 09 12:20:25.461780 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:25.461597    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls podName:3c86c595-fb8e-45e3-b6c0-cb9871af938d nodeName:}" failed. No retries permitted until 2026-09-09 12:20:25.961588748 +0000 UTC m=+46.305186679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vl2k6" (UID: "3c86c595-fb8e-45e3-b6c0-cb9871af938d") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:20:25.461780 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.461528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxzrm\" (UniqueName: \"kubernetes.io/projected/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-api-access-zxzrm\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.461780 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.461745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c86c595-fb8e-45e3-b6c0-cb9871af938d-data-volume\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.466624 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.466597    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/1.log"
Sep 09 12:20:25.467026 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.467009    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/0.log"
Sep 09 12:20:25.467113 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.467052    2578 generic.go:356] "Generic (PLEG): container finished" podID="0548208b-e765-4614-b647-363d001d99c4" containerID="b663bd8355fe598047c1146522a71123195c90bcdfb4f491fd2265b664b334a0" exitCode=255
Sep 09 12:20:25.467155 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.467133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hzzgw" event={"ID":"0548208b-e765-4614-b647-363d001d99c4","Type":"ContainerDied","Data":"b663bd8355fe598047c1146522a71123195c90bcdfb4f491fd2265b664b334a0"}
Sep 09 12:20:25.467200 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.467192    2578 scope.go:117] "RemoveContainer" containerID="48bffbc0df77c5306e7cacf519b28c4e41895ec9ac9008460b42f35af00aeb41"
Sep 09 12:20:25.467570 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.467433    2578 scope.go:117] "RemoveContainer" containerID="b663bd8355fe598047c1146522a71123195c90bcdfb4f491fd2265b664b334a0"
Sep 09 12:20:25.467678 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:25.467603    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-hzzgw_openshift-console-operator(0548208b-e765-4614-b647-363d001d99c4)\"" pod="openshift-console-operator/console-operator-85b454874b-hzzgw" podUID="0548208b-e765-4614-b647-363d001d99c4"
Sep 09 12:20:25.472028 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.472008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxzrm\" (UniqueName: \"kubernetes.io/projected/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-api-access-zxzrm\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.966823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.966782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.966977 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:25.966929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:25.966977 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:25.966938    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:20:25.967063 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:25.967018    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm podName:3c86c595-fb8e-45e3-b6c0-cb9871af938d nodeName:}" failed. No retries permitted until 2026-09-09 12:20:26.966999965 +0000 UTC m=+47.310597915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vl2k6" (UID: "3c86c595-fb8e-45e3-b6c0-cb9871af938d") : configmap "kube-rbac-proxy" not found
Sep 09 12:20:25.967063 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:25.967049    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:20:25.967131 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:25.967105    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls podName:3c86c595-fb8e-45e3-b6c0-cb9871af938d nodeName:}" failed. No retries permitted until 2026-09-09 12:20:26.967092075 +0000 UTC m=+47.310690001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vl2k6" (UID: "3c86c595-fb8e-45e3-b6c0-cb9871af938d") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:20:26.472831 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:26.472803    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/1.log"
Sep 09 12:20:26.473189 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:26.473131    2578 scope.go:117] "RemoveContainer" containerID="b663bd8355fe598047c1146522a71123195c90bcdfb4f491fd2265b664b334a0"
Sep 09 12:20:26.473313 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:26.473295    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-hzzgw_openshift-console-operator(0548208b-e765-4614-b647-363d001d99c4)\"" pod="openshift-console-operator/console-operator-85b454874b-hzzgw" podUID="0548208b-e765-4614-b647-363d001d99c4"
Sep 09 12:20:26.975202 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:26.975169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:26.975405 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:26.975263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:26.975405 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:26.975326    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:20:26.975405 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:26.975369    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:20:26.975405 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:26.975399    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls podName:3c86c595-fb8e-45e3-b6c0-cb9871af938d nodeName:}" failed. No retries permitted until 2026-09-09 12:20:28.975383518 +0000 UTC m=+49.318981458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vl2k6" (UID: "3c86c595-fb8e-45e3-b6c0-cb9871af938d") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:20:26.975634 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:26.975419    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm podName:3c86c595-fb8e-45e3-b6c0-cb9871af938d nodeName:}" failed. No retries permitted until 2026-09-09 12:20:28.975405857 +0000 UTC m=+49.319003783 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vl2k6" (UID: "3c86c595-fb8e-45e3-b6c0-cb9871af938d") : configmap "kube-rbac-proxy" not found
Sep 09 12:20:27.341848 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:27.341776    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zhgxx_b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3/dns-node-resolver/0.log"
Sep 09 12:20:28.344234 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:28.344187    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t7kpc_6bb40332-7f47-480d-8fef-7be82e907b6a/node-ca/0.log"
Sep 09 12:20:28.740438 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:28.740395    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-88dzs_d7e96ad5-f31e-45dd-b44e-e49453638364/serve-healthcheck-canary/0.log"
Sep 09 12:20:28.995617 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:28.995540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:28.995757 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:28.995629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:28.995757 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:28.995690    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:20:28.995757 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:28.995751    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:20:28.995850 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:28.995762    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls podName:3c86c595-fb8e-45e3-b6c0-cb9871af938d nodeName:}" failed. No retries permitted until 2026-09-09 12:20:32.995746222 +0000 UTC m=+53.339344148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vl2k6" (UID: "3c86c595-fb8e-45e3-b6c0-cb9871af938d") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:20:28.995850 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:28.995804    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm podName:3c86c595-fb8e-45e3-b6c0-cb9871af938d nodeName:}" failed. No retries permitted until 2026-09-09 12:20:32.995790106 +0000 UTC m=+53.339388032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vl2k6" (UID: "3c86c595-fb8e-45e3-b6c0-cb9871af938d") : configmap "kube-rbac-proxy" not found
Sep 09 12:20:30.407912 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:30.407881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:30.408307 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.408036    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 12:20:30.408307 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.408056    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4fb58b64-ghqtv: secret "image-registry-tls" not found
Sep 09 12:20:30.408307 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.408111    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls podName:4ebb9e23-a65c-4297-9e19-69bc45c4a54f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:46.408095391 +0000 UTC m=+66.751693336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls") pod "image-registry-7c4fb58b64-ghqtv" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f") : secret "image-registry-tls" not found
Sep 09 12:20:30.508930 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:30.508897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:30.509095 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:30.508957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:30.509095 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:30.509011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:30.509095 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.509070    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:30.509209 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.509107    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:20:30.509209 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.509146    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:46.50913196 +0000 UTC m=+66.852729886 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : configmap references non-existent config key: service-ca.crt
Sep 09 12:20:30.509209 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.509162    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls podName:0f9a503b-3e0d-4018-a745-aab35e92648f nodeName:}" failed. No retries permitted until 2026-09-09 12:20:46.5091549 +0000 UTC m=+66.852752825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-rc7bk" (UID: "0f9a503b-3e0d-4018-a745-aab35e92648f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:20:30.509209 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.509176    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs podName:11252896-9746-4dbf-9ca6-01c7d854fab9 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:46.509169779 +0000 UTC m=+66.852767705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs") pod "router-default-6f8c9778b-t8sr5" (UID: "11252896-9746-4dbf-9ca6-01c7d854fab9") : secret "router-metrics-certs-default" not found
Sep 09 12:20:30.609818 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:30.609779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:30.610013 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:30.609857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:30.610013 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:30.609885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:30.610013 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.609930    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:20:30.610013 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.609994    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 12:20:30.610013 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.609998    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 12:20:30.610198 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.610000    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls podName:fd502c7b-b814-4afd-8516-6d6e7fa25979 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:46.609984343 +0000 UTC m=+66.953582272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls") pod "dns-default-f95hr" (UID: "fd502c7b-b814-4afd-8516-6d6e7fa25979") : secret "dns-default-metrics-tls" not found
Sep 09 12:20:30.610198 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.610074    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls podName:449412fc-55a9-44fd-81f1-6156d47e5381 nodeName:}" failed. No retries permitted until 2026-09-09 12:20:46.610056716 +0000 UTC m=+66.953654645 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j444k" (UID: "449412fc-55a9-44fd-81f1-6156d47e5381") : secret "samples-operator-tls" not found
Sep 09 12:20:30.610198 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:30.610089    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert podName:6cb90a7b-a57f-4dc3-9231-c74b11e6173e nodeName:}" failed. No retries permitted until 2026-09-09 12:20:46.610080552 +0000 UTC m=+66.953678480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4ch9d" (UID: "6cb90a7b-a57f-4dc3-9231-c74b11e6173e") : secret "networking-console-plugin-cert" not found
Sep 09 12:20:33.028600 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:33.028568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:33.029047 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:33.028681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:33.029047 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:33.028794    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:20:33.029047 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:33.028845    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls podName:3c86c595-fb8e-45e3-b6c0-cb9871af938d nodeName:}" failed. No retries permitted until 2026-09-09 12:20:41.028832251 +0000 UTC m=+61.372430177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vl2k6" (UID: "3c86c595-fb8e-45e3-b6c0-cb9871af938d") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:20:33.029184 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:33.029059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c86c595-fb8e-45e3-b6c0-cb9871af938d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:34.956171 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:34.956135    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:34.956642 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:34.956186    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:34.956642 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:34.956521    2578 scope.go:117] "RemoveContainer" containerID="b663bd8355fe598047c1146522a71123195c90bcdfb4f491fd2265b664b334a0"
Sep 09 12:20:35.501334 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:35.501308    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:20:35.501727 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:35.501712    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/1.log"
Sep 09 12:20:35.501785 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:35.501745    2578 generic.go:356] "Generic (PLEG): container finished" podID="0548208b-e765-4614-b647-363d001d99c4" containerID="ef4ca58670775351911a9926be6c4cad8adb38fefa0a61188b0e42fabc6f2dd0" exitCode=255
Sep 09 12:20:35.501820 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:35.501801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hzzgw" event={"ID":"0548208b-e765-4614-b647-363d001d99c4","Type":"ContainerDied","Data":"ef4ca58670775351911a9926be6c4cad8adb38fefa0a61188b0e42fabc6f2dd0"}
Sep 09 12:20:35.501881 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:35.501870    2578 scope.go:117] "RemoveContainer" containerID="b663bd8355fe598047c1146522a71123195c90bcdfb4f491fd2265b664b334a0"
Sep 09 12:20:35.502184 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:35.502166    2578 scope.go:117] "RemoveContainer" containerID="ef4ca58670775351911a9926be6c4cad8adb38fefa0a61188b0e42fabc6f2dd0"
Sep 09 12:20:35.502406 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:35.502388    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-hzzgw_openshift-console-operator(0548208b-e765-4614-b647-363d001d99c4)\"" pod="openshift-console-operator/console-operator-85b454874b-hzzgw" podUID="0548208b-e765-4614-b647-363d001d99c4"
Sep 09 12:20:36.506227 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:36.506195    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:20:40.370016 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:40.369986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nvnq"
Sep 09 12:20:41.100467 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:41.100423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:41.102838 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:41.102816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c86c595-fb8e-45e3-b6c0-cb9871af938d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vl2k6\" (UID: \"3c86c595-fb8e-45e3-b6c0-cb9871af938d\") " pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:41.399414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:41.399334    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tndcj\""
Sep 09 12:20:41.408019 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:41.407996    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vl2k6"
Sep 09 12:20:41.680767 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:41.680635    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vl2k6"]
Sep 09 12:20:41.683349 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:41.683319    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c86c595_fb8e_45e3_b6c0_cb9871af938d.slice/crio-19fb330fda6c8b4d32de13a7a2c20c3d407e89d2f9c0c4c937e9dac58c1de2df WatchSource:0}: Error finding container 19fb330fda6c8b4d32de13a7a2c20c3d407e89d2f9c0c4c937e9dac58c1de2df: Status 404 returned error can't find the container with id 19fb330fda6c8b4d32de13a7a2c20c3d407e89d2f9c0c4c937e9dac58c1de2df
Sep 09 12:20:42.526344 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:42.526304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vl2k6" event={"ID":"3c86c595-fb8e-45e3-b6c0-cb9871af938d","Type":"ContainerStarted","Data":"ca2e57654a4958ff6ce13872488804dbc50debd3e94c4945b2d8053c8ba02377"}
Sep 09 12:20:42.526709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:42.526388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vl2k6" event={"ID":"3c86c595-fb8e-45e3-b6c0-cb9871af938d","Type":"ContainerStarted","Data":"d77d63c9033b1d1a389eb81cf9d27cb2ccdb2ed1c313f90fd80d1cecd5be6e1e"}
Sep 09 12:20:42.526709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:42.526407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vl2k6" event={"ID":"3c86c595-fb8e-45e3-b6c0-cb9871af938d","Type":"ContainerStarted","Data":"19fb330fda6c8b4d32de13a7a2c20c3d407e89d2f9c0c4c937e9dac58c1de2df"}
Sep 09 12:20:44.956664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:44.956628    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:44.957031 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:44.956683    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:44.957031 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:44.956997    2578 scope.go:117] "RemoveContainer" containerID="ef4ca58670775351911a9926be6c4cad8adb38fefa0a61188b0e42fabc6f2dd0"
Sep 09 12:20:44.957188 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:20:44.957169    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-hzzgw_openshift-console-operator(0548208b-e765-4614-b647-363d001d99c4)\"" pod="openshift-console-operator/console-operator-85b454874b-hzzgw" podUID="0548208b-e765-4614-b647-363d001d99c4"
Sep 09 12:20:45.539100 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:45.539063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vl2k6" event={"ID":"3c86c595-fb8e-45e3-b6c0-cb9871af938d","Type":"ContainerStarted","Data":"364bd147c35dff572cf3ef2b4615f23fa64d3fc632dedf892b23f1233b5c8422"}
Sep 09 12:20:45.559729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:45.559675    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vl2k6" podStartSLOduration=17.306904295 podStartE2EDuration="20.559657677s" totalImagesPullingTime="3.252753382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:20:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:45.558716802 +0000 UTC m=+65.902314740" watchObservedRunningTime="2026-09-09 12:20:45.559657677 +0000 UTC m=+65.903255625"
Sep 09 12:20:45.856527 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:45.856438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:45.859198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:45.859181    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 12:20:45.869444 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:45.869424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d09d81f3-b038-48e2-88c2-6ab36ddefce2-original-pull-secret\") pod \"global-pull-secret-syncer-qdgrt\" (UID: \"d09d81f3-b038-48e2-88c2-6ab36ddefce2\") " pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:45.957254 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:45.957196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:45.957644 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:45.957288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlf4\" (UniqueName: \"kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4\") pod \"network-check-target-kxk9k\" (UID: \"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610\") " pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:45.960031 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:45.960004    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 12:20:45.960031 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:45.960022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jlf4\" (UniqueName: \"kubernetes.io/projected/004699ec-e8f0-4fb3-87b1-0d5a4ff3a610-kube-api-access-8jlf4\") pod \"network-check-target-kxk9k\" (UID: \"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610\") " pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:45.970272 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:45.970242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a554455-a4e1-45a6-95d3-a129ec8da79b-metrics-certs\") pod \"network-metrics-daemon-thh9h\" (UID: \"1a554455-a4e1-45a6-95d3-a129ec8da79b\") " pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:46.092702 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.092657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdgrt"
Sep 09 12:20:46.101959 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.101930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wjmxh\""
Sep 09 12:20:46.107694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.107338    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lblnd\""
Sep 09 12:20:46.110823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.110807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh9h"
Sep 09 12:20:46.116251 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.116227    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:46.374035 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.373968    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdgrt"]
Sep 09 12:20:46.376426 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:46.376398    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09d81f3_b038_48e2_88c2_6ab36ddefce2.slice/crio-c944326bfa5f81356e9d08628ba2c60d43ddb919556ce3f773ab8c90b5aa6a69 WatchSource:0}: Error finding container c944326bfa5f81356e9d08628ba2c60d43ddb919556ce3f773ab8c90b5aa6a69: Status 404 returned error can't find the container with id c944326bfa5f81356e9d08628ba2c60d43ddb919556ce3f773ab8c90b5aa6a69
Sep 09 12:20:46.411777 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.411755    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-thh9h"]
Sep 09 12:20:46.414045 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:46.414021    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a554455_a4e1_45a6_95d3_a129ec8da79b.slice/crio-9d2ad4df09cffa984eb85d896a9e645b18701603f496e0189a263ec83299ae55 WatchSource:0}: Error finding container 9d2ad4df09cffa984eb85d896a9e645b18701603f496e0189a263ec83299ae55: Status 404 returned error can't find the container with id 9d2ad4df09cffa984eb85d896a9e645b18701603f496e0189a263ec83299ae55
Sep 09 12:20:46.436788 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.436761    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kxk9k"]
Sep 09 12:20:46.439372 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:46.439343    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004699ec_e8f0_4fb3_87b1_0d5a4ff3a610.slice/crio-48afa5046affe588fc768d01d556ba4141f32af9a37ecb004dccf6c9dd363991 WatchSource:0}: Error finding container 48afa5046affe588fc768d01d556ba4141f32af9a37ecb004dccf6c9dd363991: Status 404 returned error can't find the container with id 48afa5046affe588fc768d01d556ba4141f32af9a37ecb004dccf6c9dd363991
Sep 09 12:20:46.461267 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.461245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:46.463645 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.463625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls\") pod \"image-registry-7c4fb58b64-ghqtv\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") " pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:46.544247 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.544198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kxk9k" event={"ID":"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610","Type":"ContainerStarted","Data":"a867ff2212b11ac251bf3926a87624bad9c4f499fc45e32e8bbe43c1b552d2d6"}
Sep 09 12:20:46.544434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.544272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kxk9k" event={"ID":"004699ec-e8f0-4fb3-87b1-0d5a4ff3a610","Type":"ContainerStarted","Data":"48afa5046affe588fc768d01d556ba4141f32af9a37ecb004dccf6c9dd363991"}
Sep 09 12:20:46.544434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.544297    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:20:46.549628 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.549595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-thh9h" event={"ID":"1a554455-a4e1-45a6-95d3-a129ec8da79b","Type":"ContainerStarted","Data":"9d2ad4df09cffa984eb85d896a9e645b18701603f496e0189a263ec83299ae55"}
Sep 09 12:20:46.550839 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.550817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qdgrt" event={"ID":"d09d81f3-b038-48e2-88c2-6ab36ddefce2","Type":"ContainerStarted","Data":"c944326bfa5f81356e9d08628ba2c60d43ddb919556ce3f773ab8c90b5aa6a69"}
Sep 09 12:20:46.561748 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.561708    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kxk9k" podStartSLOduration=66.561695994 podStartE2EDuration="1m6.561695994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:46.56044275 +0000 UTC m=+66.904040697" watchObservedRunningTime="2026-09-09 12:20:46.561695994 +0000 UTC m=+66.905293941"
Sep 09 12:20:46.562191 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.562173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:46.562262 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.562243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:46.562298 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.562283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:46.562846 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.562825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11252896-9746-4dbf-9ca6-01c7d854fab9-service-ca-bundle\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:46.564730 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.564706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f9a503b-3e0d-4018-a745-aab35e92648f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-rc7bk\" (UID: \"0f9a503b-3e0d-4018-a745-aab35e92648f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:46.564730 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.564717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11252896-9746-4dbf-9ca6-01c7d854fab9-metrics-certs\") pod \"router-default-6f8c9778b-t8sr5\" (UID: \"11252896-9746-4dbf-9ca6-01c7d854fab9\") " pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:46.662902 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.662868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:46.663089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.662927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:46.663089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.662949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:46.665505 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.665477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd502c7b-b814-4afd-8516-6d6e7fa25979-metrics-tls\") pod \"dns-default-f95hr\" (UID: \"fd502c7b-b814-4afd-8516-6d6e7fa25979\") " pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:46.665734 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.665715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/449412fc-55a9-44fd-81f1-6156d47e5381-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j444k\" (UID: \"449412fc-55a9-44fd-81f1-6156d47e5381\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:46.665790 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.665713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cb90a7b-a57f-4dc3-9231-c74b11e6173e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4ch9d\" (UID: \"6cb90a7b-a57f-4dc3-9231-c74b11e6173e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:46.715827 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.715801    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bswb5\""
Sep 09 12:20:46.724431 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.724411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:46.780607 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.780584    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gcp85\""
Sep 09 12:20:46.789258 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.789241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"
Sep 09 12:20:46.795130 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.795113    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-56tvv\""
Sep 09 12:20:46.803798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.803774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:46.831026 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.830999    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hsrxf\""
Sep 09 12:20:46.840043 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.839715    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:46.846131 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.844634    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4tssv\""
Sep 09 12:20:46.853367 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.853342    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"
Sep 09 12:20:46.889370 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.889103    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xkvm4\""
Sep 09 12:20:46.896956 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:46.896927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d"
Sep 09 12:20:47.003813 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.003784    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c4fb58b64-ghqtv"]
Sep 09 12:20:47.007381 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:47.007349    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebb9e23_a65c_4297_9e19_69bc45c4a54f.slice/crio-58b8d38fb99bc7c972879b1e8e722e441fc26b6cffc4b9c702538da5f1d4d5f8 WatchSource:0}: Error finding container 58b8d38fb99bc7c972879b1e8e722e441fc26b6cffc4b9c702538da5f1d4d5f8: Status 404 returned error can't find the container with id 58b8d38fb99bc7c972879b1e8e722e441fc26b6cffc4b9c702538da5f1d4d5f8
Sep 09 12:20:47.064944 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.064919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk"]
Sep 09 12:20:47.067199 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:47.067173    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9a503b_3e0d_4018_a745_aab35e92648f.slice/crio-5d707cccfe78f6c8db8c301cd05ccaedb133193d31c6b2617252a7359c785173 WatchSource:0}: Error finding container 5d707cccfe78f6c8db8c301cd05ccaedb133193d31c6b2617252a7359c785173: Status 404 returned error can't find the container with id 5d707cccfe78f6c8db8c301cd05ccaedb133193d31c6b2617252a7359c785173
Sep 09 12:20:47.084852 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.084824    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f8c9778b-t8sr5"]
Sep 09 12:20:47.088402 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:47.088375    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11252896_9746_4dbf_9ca6_01c7d854fab9.slice/crio-1023e44f92687a1c754c7a3fca85f84f9ef246c78d6c6fdb1905102c8a657615 WatchSource:0}: Error finding container 1023e44f92687a1c754c7a3fca85f84f9ef246c78d6c6fdb1905102c8a657615: Status 404 returned error can't find the container with id 1023e44f92687a1c754c7a3fca85f84f9ef246c78d6c6fdb1905102c8a657615
Sep 09 12:20:47.124449 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.124424    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f95hr"]
Sep 09 12:20:47.130662 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:47.129935    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd502c7b_b814_4afd_8516_6d6e7fa25979.slice/crio-57931f23c7a407926b2878cda36e693c2cc32e3b3cdd07ea7c181c3a66d62f79 WatchSource:0}: Error finding container 57931f23c7a407926b2878cda36e693c2cc32e3b3cdd07ea7c181c3a66d62f79: Status 404 returned error can't find the container with id 57931f23c7a407926b2878cda36e693c2cc32e3b3cdd07ea7c181c3a66d62f79
Sep 09 12:20:47.145971 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.145940    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k"]
Sep 09 12:20:47.197540 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.197453    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-4ch9d"]
Sep 09 12:20:47.559116 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.559027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f8c9778b-t8sr5" event={"ID":"11252896-9746-4dbf-9ca6-01c7d854fab9","Type":"ContainerStarted","Data":"05b7fb3c847e34305ad6142ca621e4d7c992fb4359d1707e43a35a087d9539b8"}
Sep 09 12:20:47.559116 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.559083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f8c9778b-t8sr5" event={"ID":"11252896-9746-4dbf-9ca6-01c7d854fab9","Type":"ContainerStarted","Data":"1023e44f92687a1c754c7a3fca85f84f9ef246c78d6c6fdb1905102c8a657615"}
Sep 09 12:20:47.561698 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.561644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk" event={"ID":"0f9a503b-3e0d-4018-a745-aab35e92648f","Type":"ContainerStarted","Data":"5d707cccfe78f6c8db8c301cd05ccaedb133193d31c6b2617252a7359c785173"}
Sep 09 12:20:47.565484 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.564837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv" event={"ID":"4ebb9e23-a65c-4297-9e19-69bc45c4a54f","Type":"ContainerStarted","Data":"65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d"}
Sep 09 12:20:47.565484 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.564888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv" event={"ID":"4ebb9e23-a65c-4297-9e19-69bc45c4a54f","Type":"ContainerStarted","Data":"58b8d38fb99bc7c972879b1e8e722e441fc26b6cffc4b9c702538da5f1d4d5f8"}
Sep 09 12:20:47.565484 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.565481    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:20:47.567112 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.567041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d" event={"ID":"6cb90a7b-a57f-4dc3-9231-c74b11e6173e","Type":"ContainerStarted","Data":"2669d8b22fd1af59f0cc7f68a13eb2147297a58b81b883f9922eeea12ba4fd58"}
Sep 09 12:20:47.570553 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.570461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k" event={"ID":"449412fc-55a9-44fd-81f1-6156d47e5381","Type":"ContainerStarted","Data":"5bd819454673b6c8a8dfff9366db1828191c173b41f2f3e980adbeccf6ecd328"}
Sep 09 12:20:47.573943 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.573920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f95hr" event={"ID":"fd502c7b-b814-4afd-8516-6d6e7fa25979","Type":"ContainerStarted","Data":"57931f23c7a407926b2878cda36e693c2cc32e3b3cdd07ea7c181c3a66d62f79"}
Sep 09 12:20:47.579692 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.579655    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6f8c9778b-t8sr5" podStartSLOduration=45.579641776 podStartE2EDuration="45.579641776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:47.579019022 +0000 UTC m=+67.922616971" watchObservedRunningTime="2026-09-09 12:20:47.579641776 +0000 UTC m=+67.923239718"
Sep 09 12:20:47.597625 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.597577    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv" podStartSLOduration=67.59755859 podStartE2EDuration="1m7.59755859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:47.5969096 +0000 UTC m=+67.940507548" watchObservedRunningTime="2026-09-09 12:20:47.59755859 +0000 UTC m=+67.941156538"
Sep 09 12:20:47.804772 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.804570    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:47.807718 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:47.807697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:48.587799 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:48.587761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-thh9h" event={"ID":"1a554455-a4e1-45a6-95d3-a129ec8da79b","Type":"ContainerStarted","Data":"2ccd24f9f9834a8ac85ab079c3c77ec0ad8c578e8530cf4cdac22e4c4d206ef2"}
Sep 09 12:20:48.588534 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:48.588513    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:48.589289 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:48.588572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-thh9h" event={"ID":"1a554455-a4e1-45a6-95d3-a129ec8da79b","Type":"ContainerStarted","Data":"36c943fdcbdddbd247b90e316167221a765527a50af8646dc9bec00e5dfd2d64"}
Sep 09 12:20:48.589634 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:48.589598    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6f8c9778b-t8sr5"
Sep 09 12:20:48.606838 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:48.606780    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-thh9h" podStartSLOduration=67.239673367 podStartE2EDuration="1m8.606759355s" totalImagesPullingTime="1.367085988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:48.604383577 +0000 UTC m=+68.947981528" watchObservedRunningTime="2026-09-09 12:20:48.606759355 +0000 UTC m=+68.950357304"
Sep 09 12:20:52.604167 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.604112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk" event={"ID":"0f9a503b-3e0d-4018-a745-aab35e92648f","Type":"ContainerStarted","Data":"5ccb5228a73341ba05715e672ff66cff714c743d36a7550e698b4ab497843c60"}
Sep 09 12:20:52.608143 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.608116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d" event={"ID":"6cb90a7b-a57f-4dc3-9231-c74b11e6173e","Type":"ContainerStarted","Data":"cf27e87044cf0c9490654250a2f98dedba7c14bc9e0ef8e796cb6963941aef7a"}
Sep 09 12:20:52.610713 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.610544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k" event={"ID":"449412fc-55a9-44fd-81f1-6156d47e5381","Type":"ContainerStarted","Data":"e847aa5d6a2e6a898ca7b8027139508819575624dc7d0c276162afdd0b07506f"}
Sep 09 12:20:52.610713 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.610596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k" event={"ID":"449412fc-55a9-44fd-81f1-6156d47e5381","Type":"ContainerStarted","Data":"128e007c4087b900ac97c3399cf27543f7eacf5eedab390de7f936d1f593f946"}
Sep 09 12:20:52.613080 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.613004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f95hr" event={"ID":"fd502c7b-b814-4afd-8516-6d6e7fa25979","Type":"ContainerStarted","Data":"eac1cd1f1d80e67489b885064d90e753518fd44e2a7ea8f818852fb71f8dc118"}
Sep 09 12:20:52.620936 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.620651    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-rc7bk" podStartSLOduration=45.55012805 podStartE2EDuration="50.620636609s" totalImagesPullingTime="5.070508559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:52.620002816 +0000 UTC m=+72.963600781" watchObservedRunningTime="2026-09-09 12:20:52.620636609 +0000 UTC m=+72.964234557"
Sep 09 12:20:52.641076 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.640693    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j444k" podStartSLOduration=45.716085619 podStartE2EDuration="50.640672958s" totalImagesPullingTime="4.924587339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:52.638855442 +0000 UTC m=+72.982453409" watchObservedRunningTime="2026-09-09 12:20:52.640672958 +0000 UTC m=+72.984270908"
Sep 09 12:20:52.660062 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.660016    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-4ch9d" podStartSLOduration=40.734034465 podStartE2EDuration="45.659995158s" totalImagesPullingTime="4.925960693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:52.659720208 +0000 UTC m=+73.003318156" watchObservedRunningTime="2026-09-09 12:20:52.659995158 +0000 UTC m=+73.003593106"
Sep 09 12:20:52.804503 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.804474    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4"]
Sep 09 12:20:52.829809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.829778    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4"]
Sep 09 12:20:52.829962 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.829875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4"
Sep 09 12:20:52.831937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.831916    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 12:20:52.832059 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.831974    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z49b5\""
Sep 09 12:20:52.922023 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:52.921984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b2e52117-b695-4db3-95e1-be46d4f2a998-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-cn2x4\" (UID: \"b2e52117-b695-4db3-95e1-be46d4f2a998\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4"
Sep 09 12:20:53.022394 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:53.022358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b2e52117-b695-4db3-95e1-be46d4f2a998-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-cn2x4\" (UID: \"b2e52117-b695-4db3-95e1-be46d4f2a998\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4"
Sep 09 12:20:53.024946 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:53.024926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b2e52117-b695-4db3-95e1-be46d4f2a998-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-cn2x4\" (UID: \"b2e52117-b695-4db3-95e1-be46d4f2a998\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4"
Sep 09 12:20:53.141285 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:53.141249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4"
Sep 09 12:20:53.431109 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:53.431079    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4"]
Sep 09 12:20:53.434354 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:20:53.434328    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e52117_b695_4db3_95e1_be46d4f2a998.slice/crio-5ec66a7cf234604ddb7c743b6683cce867dd57402d6647dc9c80009cd021d520 WatchSource:0}: Error finding container 5ec66a7cf234604ddb7c743b6683cce867dd57402d6647dc9c80009cd021d520: Status 404 returned error can't find the container with id 5ec66a7cf234604ddb7c743b6683cce867dd57402d6647dc9c80009cd021d520
Sep 09 12:20:53.619075 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:53.618983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qdgrt" event={"ID":"d09d81f3-b038-48e2-88c2-6ab36ddefce2","Type":"ContainerStarted","Data":"c7181b10363cf03db1a67888418b279c29c663b91da35edd09d128a45ac6dac4"}
Sep 09 12:20:53.620251 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:53.620205    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4" event={"ID":"b2e52117-b695-4db3-95e1-be46d4f2a998","Type":"ContainerStarted","Data":"5ec66a7cf234604ddb7c743b6683cce867dd57402d6647dc9c80009cd021d520"}
Sep 09 12:20:53.621925 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:53.621903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f95hr" event={"ID":"fd502c7b-b814-4afd-8516-6d6e7fa25979","Type":"ContainerStarted","Data":"c20e9fadc9e0990b408bfa385f75ce76c774b44a7909ef416bc7479c1f9a9a57"}
Sep 09 12:20:53.636039 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:53.635990    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qdgrt" podStartSLOduration=67.511037295 podStartE2EDuration="1m13.635965382s" totalImagesPullingTime="6.124928087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:53.63531539 +0000 UTC m=+73.978913338" watchObservedRunningTime="2026-09-09 12:20:53.635965382 +0000 UTC m=+73.979563330"
Sep 09 12:20:53.652161 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:53.652103    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-f95hr" podStartSLOduration=34.649159492 podStartE2EDuration="39.652083109s" totalImagesPullingTime="5.002923617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:53.651182022 +0000 UTC m=+73.994779980" watchObservedRunningTime="2026-09-09 12:20:53.652083109 +0000 UTC m=+73.995681058"
Sep 09 12:20:54.625331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:54.625300    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f95hr"
Sep 09 12:20:55.631177 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:55.631137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4" event={"ID":"b2e52117-b695-4db3-95e1-be46d4f2a998","Type":"ContainerStarted","Data":"55084902a49585531dab5f13398c691e73d53f2c0f3c4dc11c1e92d8d496d9c9"}
Sep 09 12:20:55.631618 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:55.631413    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4"
Sep 09 12:20:55.635988 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:55.635967    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4"
Sep 09 12:20:55.648520 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:55.648475    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cn2x4" podStartSLOduration=2.108551668 podStartE2EDuration="3.64846176s" totalImagesPullingTime="1.539910092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:55.646672341 +0000 UTC m=+75.990270288" watchObservedRunningTime="2026-09-09 12:20:55.64846176 +0000 UTC m=+75.992059717"
Sep 09 12:20:58.180167 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:58.180137    2578 scope.go:117] "RemoveContainer" containerID="ef4ca58670775351911a9926be6c4cad8adb38fefa0a61188b0e42fabc6f2dd0"
Sep 09 12:20:58.643438 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:58.643356    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:20:58.643438 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:58.643418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hzzgw" event={"ID":"0548208b-e765-4614-b647-363d001d99c4","Type":"ContainerStarted","Data":"6982cce7a62d681e2ff4b258c774974f0d26e159c0a5a0d20ceb5817d709d61c"}
Sep 09 12:20:58.643696 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:58.643680    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:20:58.660839 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:58.660790    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-hzzgw" podStartSLOduration=48.774581171 podStartE2EDuration="56.660776118s" totalImagesPullingTime="7.886194947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:58.660130489 +0000 UTC m=+79.003728436" watchObservedRunningTime="2026-09-09 12:20:58.660776118 +0000 UTC m=+79.004374065"
Sep 09 12:20:58.790521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:20:58.790495    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-hzzgw"
Sep 09 12:21:01.275710 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.275673    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qm2bh"]
Sep 09 12:21:01.278066 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.278043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.280168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.280130    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 12:21:01.280168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.280157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tnpqb\""
Sep 09 12:21:01.280760 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.280739    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 12:21:01.280866 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.280803    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 12:21:01.280955 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.280936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 12:21:01.387723 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.387607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-tls\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.387723 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.387665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-wtmp\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.387723 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.387696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.387987 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.387722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.387987 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.387754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b593e-ca9f-415e-884d-80d63e65d5e1-metrics-client-ca\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.387987 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.387771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c1b593e-ca9f-415e-884d-80d63e65d5e1-root\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.387987 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.387876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-textfile\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.387987 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.387906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbxtw\" (UniqueName: \"kubernetes.io/projected/9c1b593e-ca9f-415e-884d-80d63e65d5e1-kube-api-access-bbxtw\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.387987 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.387925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c1b593e-ca9f-415e-884d-80d63e65d5e1-sys\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.488690 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.488648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b593e-ca9f-415e-884d-80d63e65d5e1-metrics-client-ca\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.488871 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.488697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c1b593e-ca9f-415e-884d-80d63e65d5e1-root\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.488871 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.488746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-textfile\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.488871 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.488775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxtw\" (UniqueName: \"kubernetes.io/projected/9c1b593e-ca9f-415e-884d-80d63e65d5e1-kube-api-access-bbxtw\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.488871 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.488797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c1b593e-ca9f-415e-884d-80d63e65d5e1-sys\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.488871 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.488822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-tls\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.488871 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.488840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c1b593e-ca9f-415e-884d-80d63e65d5e1-root\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.489166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.488872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c1b593e-ca9f-415e-884d-80d63e65d5e1-sys\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.489166 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:21:01.488942    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 12:21:01.489166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.488941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-wtmp\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.489166 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:21:01.489000    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-tls podName:9c1b593e-ca9f-415e-884d-80d63e65d5e1 nodeName:}" failed. No retries permitted until 2026-09-09 12:21:01.988978516 +0000 UTC m=+82.332576465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-tls") pod "node-exporter-qm2bh" (UID: "9c1b593e-ca9f-415e-884d-80d63e65d5e1") : secret "node-exporter-tls" not found
Sep 09 12:21:01.489166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.489028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.489166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.489045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.489166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.489059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-wtmp\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.489166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.489094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-textfile\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.489602 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.489335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b593e-ca9f-415e-884d-80d63e65d5e1-metrics-client-ca\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.489656 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.489595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.491659 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.491629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.497191 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.497167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxtw\" (UniqueName: \"kubernetes.io/projected/9c1b593e-ca9f-415e-884d-80d63e65d5e1-kube-api-access-bbxtw\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.993297 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.993261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-tls\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:01.995626 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:01.995605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c1b593e-ca9f-415e-884d-80d63e65d5e1-node-exporter-tls\") pod \"node-exporter-qm2bh\" (UID: \"9c1b593e-ca9f-415e-884d-80d63e65d5e1\") " pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:02.187684 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.187653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qm2bh"
Sep 09 12:21:02.199453 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:21:02.199424    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1b593e_ca9f_415e_884d_80d63e65d5e1.slice/crio-dc63216252aa6568b81a7d3957db51ac20804b3e60a720016745c3ceff05d27c WatchSource:0}: Error finding container dc63216252aa6568b81a7d3957db51ac20804b3e60a720016745c3ceff05d27c: Status 404 returned error can't find the container with id dc63216252aa6568b81a7d3957db51ac20804b3e60a720016745c3ceff05d27c
Sep 09 12:21:02.329499 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.329415    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:21:02.332492 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.332473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.334450 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.334421    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 12:21:02.334450 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.334433    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 12:21:02.334608 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.334556    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 12:21:02.334608 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.334562    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 12:21:02.334608 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.334569    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 12:21:02.334785 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.334769    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 12:21:02.334883 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.334863    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 12:21:02.335065 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.335046    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-24vfz\""
Sep 09 12:21:02.335158 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.335063    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 12:21:02.335158 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.335082    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 12:21:02.345021 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.345001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:21:02.397134 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlbb\" (UniqueName: \"kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-kube-api-access-xnlbb\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397134 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397420 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397420 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397420 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-web-config\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397420 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397420 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397420 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397420 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397662 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397662 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397662 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-out\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.397662 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.397582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.498566 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.498566 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.498808 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.498808 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.498808 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-out\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.498808 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.498808 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlbb\" (UniqueName: \"kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-kube-api-access-xnlbb\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.498808 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.498808 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.498808 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.499187 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-web-config\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.499187 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.499187 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.498865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.499454 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.499331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.499701 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:21:02.499680    2578 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 12:21:02.499768 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:21:02.499753    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-main-tls podName:e2effd5c-94a3-411b-8bd7-ddd419281f7b nodeName:}" failed. No retries permitted until 2026-09-09 12:21:02.999730987 +0000 UTC m=+83.343328927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b") : secret "alertmanager-main-tls" not found
Sep 09 12:21:02.499862 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:21:02.499847    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-trusted-ca-bundle podName:e2effd5c-94a3-411b-8bd7-ddd419281f7b nodeName:}" failed. No retries permitted until 2026-09-09 12:21:02.999833585 +0000 UTC m=+83.343431511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b") : configmap references non-existent config key: ca-bundle.crt
Sep 09 12:21:02.500604 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.500551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.501791 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.501739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.501890 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.501788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.502025 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.502004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.502297 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.502280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-out\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.502670 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.502651    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.502774 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.502753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.504046 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.504028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-web-config\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.504297 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.504281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.507297 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.507278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlbb\" (UniqueName: \"kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-kube-api-access-xnlbb\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:02.660151 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:02.660119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm2bh" event={"ID":"9c1b593e-ca9f-415e-884d-80d63e65d5e1","Type":"ContainerStarted","Data":"dc63216252aa6568b81a7d3957db51ac20804b3e60a720016745c3ceff05d27c"}
Sep 09 12:21:03.003445 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.003358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:03.003445 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.003404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:03.004283 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.004259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:03.006638 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.006611    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:03.241777 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.241740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:21:03.243887 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.243748    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"]
Sep 09 12:21:03.246928 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.246908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.249355 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.249103    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 12:21:03.249355 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.249116    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-17ajikkl5s1is\""
Sep 09 12:21:03.249355 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.249161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 12:21:03.249355 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.249161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-q2hjj\""
Sep 09 12:21:03.249355 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.249239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 12:21:03.249355 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.249247    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 12:21:03.249355 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.249288    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 12:21:03.260541 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.260515    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"]
Sep 09 12:21:03.407132 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.407099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-grpc-tls\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.407715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.407149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.407715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.407182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww9gt\" (UniqueName: \"kubernetes.io/projected/ff25c9f1-6d1d-438b-affd-f726ea7e7281-kube-api-access-ww9gt\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.407715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.407229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.407715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.407285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.407715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.407350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-tls\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.407715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.407372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff25c9f1-6d1d-438b-affd-f726ea7e7281-metrics-client-ca\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.407715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.407442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.508160 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.508085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww9gt\" (UniqueName: \"kubernetes.io/projected/ff25c9f1-6d1d-438b-affd-f726ea7e7281-kube-api-access-ww9gt\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.508160 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.508130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.508160 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.508168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.508459 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.508207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-tls\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.508459 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.508257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff25c9f1-6d1d-438b-affd-f726ea7e7281-metrics-client-ca\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.508459 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.508326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.508459 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.508365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-grpc-tls\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.508459 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.508394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.509373 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.509315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff25c9f1-6d1d-438b-affd-f726ea7e7281-metrics-client-ca\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.511294 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.511248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-tls\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.511409 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.511379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.511470 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.511403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.511470 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.511458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.511785 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.511765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.511967 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.511944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff25c9f1-6d1d-438b-affd-f726ea7e7281-secret-grpc-tls\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.515930 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.515905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww9gt\" (UniqueName: \"kubernetes.io/projected/ff25c9f1-6d1d-438b-affd-f726ea7e7281-kube-api-access-ww9gt\") pod \"thanos-querier-689c6d9ddd-rn82x\" (UID: \"ff25c9f1-6d1d-438b-affd-f726ea7e7281\") " pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.538195 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.538170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:21:03.540783 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:21:03.540761    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2effd5c_94a3_411b_8bd7_ddd419281f7b.slice/crio-a60eb5d6df3fc517ce60f4d52c2ce7d6fdfac4bb073425a14ac69043789d201b WatchSource:0}: Error finding container a60eb5d6df3fc517ce60f4d52c2ce7d6fdfac4bb073425a14ac69043789d201b: Status 404 returned error can't find the container with id a60eb5d6df3fc517ce60f4d52c2ce7d6fdfac4bb073425a14ac69043789d201b
Sep 09 12:21:03.557767 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.557742    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:03.667917 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.667569    2578 generic.go:356] "Generic (PLEG): container finished" podID="9c1b593e-ca9f-415e-884d-80d63e65d5e1" containerID="d9c25960d41fe3ae411f5e6a3b034b7b7c986cfee220f7a39bd3a5245e716878" exitCode=0
Sep 09 12:21:03.667917 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.667695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm2bh" event={"ID":"9c1b593e-ca9f-415e-884d-80d63e65d5e1","Type":"ContainerDied","Data":"d9c25960d41fe3ae411f5e6a3b034b7b7c986cfee220f7a39bd3a5245e716878"}
Sep 09 12:21:03.669804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.669774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerStarted","Data":"a60eb5d6df3fc517ce60f4d52c2ce7d6fdfac4bb073425a14ac69043789d201b"}
Sep 09 12:21:03.830377 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:03.830054    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"]
Sep 09 12:21:03.832890 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:21:03.832865    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff25c9f1_6d1d_438b_affd_f726ea7e7281.slice/crio-ecdc37b075db1b71110c57d23c238f168235a6cd98d53cc561ce69d10b2f9b1a WatchSource:0}: Error finding container ecdc37b075db1b71110c57d23c238f168235a6cd98d53cc561ce69d10b2f9b1a: Status 404 returned error can't find the container with id ecdc37b075db1b71110c57d23c238f168235a6cd98d53cc561ce69d10b2f9b1a
Sep 09 12:21:04.634435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:04.634404    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f95hr"
Sep 09 12:21:04.679873 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:04.679838    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm2bh" event={"ID":"9c1b593e-ca9f-415e-884d-80d63e65d5e1","Type":"ContainerStarted","Data":"cb7bd2f5e81a94325ad4a78b52689bdca14a3aab3430715e5089706649f2f04b"}
Sep 09 12:21:04.680041 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:04.679897    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm2bh" event={"ID":"9c1b593e-ca9f-415e-884d-80d63e65d5e1","Type":"ContainerStarted","Data":"6f1f059d4de5eeef2452f8d11a40842146fbf45d6f58a7b64d0415310de9155e"}
Sep 09 12:21:04.681750 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:04.681720    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x" event={"ID":"ff25c9f1-6d1d-438b-affd-f726ea7e7281","Type":"ContainerStarted","Data":"ecdc37b075db1b71110c57d23c238f168235a6cd98d53cc561ce69d10b2f9b1a"}
Sep 09 12:21:04.698258 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:04.698189    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qm2bh" podStartSLOduration=2.5462874639999997 podStartE2EDuration="3.698167475s" totalImagesPullingTime="1.084109371s" totalInitContainerRuntime="67.77064ms" isStatefulPod=true podCreationTimestamp="2026-09-09 12:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:21:04.697613537 +0000 UTC m=+85.041211486" watchObservedRunningTime="2026-09-09 12:21:04.698167475 +0000 UTC m=+85.041765423"
Sep 09 12:21:05.565082 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.565051    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6c9c7b7756-5smfc"]
Sep 09 12:21:05.567564 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.567531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.569941 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.569912    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bfkvsqoc0pop8\""
Sep 09 12:21:05.570441 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.570422    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 12:21:05.570540 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.570451    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-47hh9\""
Sep 09 12:21:05.570540 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.570508    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 12:21:05.570751 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.570735    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 12:21:05.570878 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.570861    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 12:21:05.579041 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.579018    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c9c7b7756-5smfc"]
Sep 09 12:21:05.689072 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.689038    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerID="f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e" exitCode=0
Sep 09 12:21:05.689544 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.689119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerDied","Data":"f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e"}
Sep 09 12:21:05.728648 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.728616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/438d1315-9ce0-40cc-8238-fda2188b0eba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.728809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.728671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/438d1315-9ce0-40cc-8238-fda2188b0eba-audit-log\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.728809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.728733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/438d1315-9ce0-40cc-8238-fda2188b0eba-client-ca-bundle\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.728809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.728791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rmw6\" (UniqueName: \"kubernetes.io/projected/438d1315-9ce0-40cc-8238-fda2188b0eba-kube-api-access-2rmw6\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.728933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.728858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/438d1315-9ce0-40cc-8238-fda2188b0eba-secret-metrics-server-client-certs\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.728933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.728910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/438d1315-9ce0-40cc-8238-fda2188b0eba-metrics-server-audit-profiles\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.729004 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.728945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/438d1315-9ce0-40cc-8238-fda2188b0eba-secret-metrics-server-tls\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.829985 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.829902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/438d1315-9ce0-40cc-8238-fda2188b0eba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.830156 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.830003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/438d1315-9ce0-40cc-8238-fda2188b0eba-audit-log\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.830156 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.830065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/438d1315-9ce0-40cc-8238-fda2188b0eba-client-ca-bundle\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.830156 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.830117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rmw6\" (UniqueName: \"kubernetes.io/projected/438d1315-9ce0-40cc-8238-fda2188b0eba-kube-api-access-2rmw6\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.830347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.830185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/438d1315-9ce0-40cc-8238-fda2188b0eba-secret-metrics-server-client-certs\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.830347 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.830308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/438d1315-9ce0-40cc-8238-fda2188b0eba-metrics-server-audit-profiles\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.830448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.830376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/438d1315-9ce0-40cc-8238-fda2188b0eba-secret-metrics-server-tls\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.830669 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.830644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/438d1315-9ce0-40cc-8238-fda2188b0eba-audit-log\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.830769 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.830716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/438d1315-9ce0-40cc-8238-fda2188b0eba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.831529 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.831503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/438d1315-9ce0-40cc-8238-fda2188b0eba-metrics-server-audit-profiles\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.833297 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.833270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/438d1315-9ce0-40cc-8238-fda2188b0eba-secret-metrics-server-client-certs\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.833608 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.833588    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/438d1315-9ce0-40cc-8238-fda2188b0eba-client-ca-bundle\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.833608 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.833598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/438d1315-9ce0-40cc-8238-fda2188b0eba-secret-metrics-server-tls\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.838917 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.838898    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rmw6\" (UniqueName: \"kubernetes.io/projected/438d1315-9ce0-40cc-8238-fda2188b0eba-kube-api-access-2rmw6\") pod \"metrics-server-6c9c7b7756-5smfc\" (UID: \"438d1315-9ce0-40cc-8238-fda2188b0eba\") " pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:05.879950 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:05.879925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:06.268250 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:06.268200    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c9c7b7756-5smfc"]
Sep 09 12:21:06.271921 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:21:06.271885    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438d1315_9ce0_40cc_8238_fda2188b0eba.slice/crio-c08ca0a3ec1aed895ce87030813a0c9038611d1c15049a7b460936743fd21736 WatchSource:0}: Error finding container c08ca0a3ec1aed895ce87030813a0c9038611d1c15049a7b460936743fd21736: Status 404 returned error can't find the container with id c08ca0a3ec1aed895ce87030813a0c9038611d1c15049a7b460936743fd21736
Sep 09 12:21:06.698650 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:06.698618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x" event={"ID":"ff25c9f1-6d1d-438b-affd-f726ea7e7281","Type":"ContainerStarted","Data":"411c3051df73aba7ba054290b067ac5bb01668951b8e133b7894c974d3ea4921"}
Sep 09 12:21:06.699052 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:06.698685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x" event={"ID":"ff25c9f1-6d1d-438b-affd-f726ea7e7281","Type":"ContainerStarted","Data":"2010dee3400e227681cf59dcbcd70a74cefafff145347f58df03443b7a758590"}
Sep 09 12:21:06.699052 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:06.698712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x" event={"ID":"ff25c9f1-6d1d-438b-affd-f726ea7e7281","Type":"ContainerStarted","Data":"1a844ec9166c9960e0feec1d6b3a498dcf7beca5a4f2def1b5aa0b8b1e1e12a5"}
Sep 09 12:21:06.699837 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:06.699813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc" event={"ID":"438d1315-9ce0-40cc-8238-fda2188b0eba","Type":"ContainerStarted","Data":"c08ca0a3ec1aed895ce87030813a0c9038611d1c15049a7b460936743fd21736"}
Sep 09 12:21:06.729307 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:06.729270    2578 patch_prober.go:28] interesting pod/image-registry-7c4fb58b64-ghqtv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 12:21:06.729449 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:06.729325    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv" podUID="4ebb9e23-a65c-4297-9e19-69bc45c4a54f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 12:21:07.510129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.510094    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:21:07.513083 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.513062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.515398 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.515372    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 12:21:07.515512 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.515483    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 12:21:07.515693 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.515656    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cx6vv\""
Sep 09 12:21:07.515813 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.515732    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 12:21:07.515813 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.515755    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 12:21:07.515813 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.515796    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 12:21:07.515969 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.515953    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 12:21:07.516020 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.515968    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 12:21:07.516129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.516110    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9aefhn32l0u95\""
Sep 09 12:21:07.516201 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.516173    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 12:21:07.516454 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.516426    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 12:21:07.516554 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.516475    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 12:21:07.516656 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.516431    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 12:21:07.516750 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.516698    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 12:21:07.516964 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.516937    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 12:21:07.518279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.518263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 12:21:07.531563 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.531529    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:21:07.646736 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.646641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.646736 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.646707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6w7\" (UniqueName: \"kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-kube-api-access-qx6w7\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.646945 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.646777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.646945 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.646800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.646945 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.646825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.646945 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.646848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.646945 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.646883    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.646945 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.646906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647122 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.646982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647122 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.647019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-config\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647122 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.647066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647122 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.647083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647122 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.647109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647356 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.647133    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647356 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.647158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647356 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.647196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647356 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.647273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647356 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.647313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647595 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.647388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.647595 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.647415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.709135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.709102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerStarted","Data":"c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec"}
Sep 09 12:21:07.709480 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.709168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerStarted","Data":"8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25"}
Sep 09 12:21:07.709480 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.709195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerStarted","Data":"f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1"}
Sep 09 12:21:07.709480 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.709239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerStarted","Data":"9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05"}
Sep 09 12:21:07.709480 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.709265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerStarted","Data":"235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c"}
Sep 09 12:21:07.748878 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.748847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.748986 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.748906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6w7\" (UniqueName: \"kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-kube-api-access-qx6w7\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.748986 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.748953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.748986 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.748975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749110 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.748997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749110 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749110 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749110 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-config\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749507 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749507 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749507 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749507 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749507 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749507 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749507 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.749842 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.749704    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.750325 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.750302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.751554 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.751468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.752559 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.752532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.754172 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.753862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.754172 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.753885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.754172 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.753893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.754569 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.754373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.755444 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.755419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.755737 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.755685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.755737 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.755700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.756793 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.756329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-config\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.756793 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.756419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.756793 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.756724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.756793 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.756744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.757087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.757064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.757506 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.757465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.757840 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.757818    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.758482 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.758445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.761800 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.761778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6w7\" (UniqueName: \"kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-kube-api-access-qx6w7\") pod \"prometheus-k8s-0\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:07.825363 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:07.825335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:08.117596 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.117547    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:21:08.448092 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:21:08.448062    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b49a95_daf0_4522_b783_a259fa6213fb.slice/crio-b01f34ecdcf739ddc67ef27ca24da7a8faa2476d2cc6f2cee8b0f7b4c9e80c94 WatchSource:0}: Error finding container b01f34ecdcf739ddc67ef27ca24da7a8faa2476d2cc6f2cee8b0f7b4c9e80c94: Status 404 returned error can't find the container with id b01f34ecdcf739ddc67ef27ca24da7a8faa2476d2cc6f2cee8b0f7b4c9e80c94
Sep 09 12:21:08.716715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.716619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerStarted","Data":"3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951"}
Sep 09 12:21:08.720370 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.720341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x" event={"ID":"ff25c9f1-6d1d-438b-affd-f726ea7e7281","Type":"ContainerStarted","Data":"bd1636a156669f7cee096f27fd7b723339e473b5f432ecdaa692790a1db1a987"}
Sep 09 12:21:08.720517 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.720421    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:08.720517 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.720464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x" event={"ID":"ff25c9f1-6d1d-438b-affd-f726ea7e7281","Type":"ContainerStarted","Data":"c83e8f70c9c08891f77acbb5a9f6ff9ecec00620f692f2478213dfa337f0a320"}
Sep 09 12:21:08.720517 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.720501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x" event={"ID":"ff25c9f1-6d1d-438b-affd-f726ea7e7281","Type":"ContainerStarted","Data":"e1d2dcfe011bada586f4077f0d68ae3bd00900a80307c9395768b4909dc0d8ff"}
Sep 09 12:21:08.722071 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.722047    2578 generic.go:356] "Generic (PLEG): container finished" podID="99b49a95-daf0-4522-b783-a259fa6213fb" containerID="2cc9b80db7c8a031750758966a87cfc437ef2076991466446ab6e9637d779aee" exitCode=0
Sep 09 12:21:08.722185 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.722125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerDied","Data":"2cc9b80db7c8a031750758966a87cfc437ef2076991466446ab6e9637d779aee"}
Sep 09 12:21:08.722277 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.722199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerStarted","Data":"b01f34ecdcf739ddc67ef27ca24da7a8faa2476d2cc6f2cee8b0f7b4c9e80c94"}
Sep 09 12:21:08.723813 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.723787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc" event={"ID":"438d1315-9ce0-40cc-8238-fda2188b0eba","Type":"ContainerStarted","Data":"a2e37962a04cdfe11525ad138e7ee3322779bf0559b4c4dbcdd2b229ee5e2b35"}
Sep 09 12:21:08.743238 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.743124    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.014420281 podStartE2EDuration="6.743105318s" totalImagesPullingTime="3.710945014s" totalInitContainerRuntime="17.740023ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:21:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:21:08.741515187 +0000 UTC m=+89.085113136" watchObservedRunningTime="2026-09-09 12:21:08.743105318 +0000 UTC m=+89.086703268"
Sep 09 12:21:08.758450 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.758365    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc" podStartSLOduration=1.520197588 podStartE2EDuration="3.758345597s" totalImagesPullingTime="2.238148009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:21:08.757966872 +0000 UTC m=+89.101564821" watchObservedRunningTime="2026-09-09 12:21:08.758345597 +0000 UTC m=+89.101943549"
Sep 09 12:21:08.781002 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:08.780958    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x" podStartSLOduration=2.075165868 podStartE2EDuration="5.780942761s" totalImagesPullingTime="3.705776893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:21:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:21:08.779230936 +0000 UTC m=+89.122828883" watchObservedRunningTime="2026-09-09 12:21:08.780942761 +0000 UTC m=+89.124540709"
Sep 09 12:21:09.596619 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:09.596588    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:21:12.750563 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:12.750524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerStarted","Data":"6e9d6e5f19126d49f1c6f038f0ec749b212c7f095c6bda6d04a52b994b28952a"}
Sep 09 12:21:12.750961 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:12.750601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerStarted","Data":"642b8a813c3b377bfdbedae57b66581865b10bbd2689d564e42650b2200ac740"}
Sep 09 12:21:12.750961 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:12.750627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerStarted","Data":"425efd287744b767f825ceaae0d81826aec51bc724f4f540f35749604eec7c4f"}
Sep 09 12:21:12.750961 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:12.750651    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerStarted","Data":"9d4cdf40a97c7572ec40ce580efa2a27d89f8d1ae17344e59a359d13a21bc34d"}
Sep 09 12:21:13.758016 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:13.757978    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerStarted","Data":"1fc6aef40d49ffe2368a7f9d1cb5315d6644a783ef57d945fbabcec6d486cd32"}
Sep 09 12:21:13.758430 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:13.758090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerStarted","Data":"65b59f2fd8396d2c4557e9c14c2bed3f2ae8ed3e8dceddc62f7648064e28c6c5"}
Sep 09 12:21:13.788025 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:13.787966    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.051023097 podStartE2EDuration="6.787944919s" totalImagesPullingTime="3.711006071s" totalInitContainerRuntime="25.915751ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:21:13.784970333 +0000 UTC m=+94.128568282" watchObservedRunningTime="2026-09-09 12:21:13.787944919 +0000 UTC m=+94.131542869"
Sep 09 12:21:14.171262 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:14.171226    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c4fb58b64-ghqtv"]
Sep 09 12:21:14.734882 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:14.734857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-689c6d9ddd-rn82x"
Sep 09 12:21:17.577206 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:17.577169    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kxk9k"
Sep 09 12:21:17.826147 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:17.826110    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:21:25.880232 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:25.880181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:25.880625 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:25.880265    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:39.192881 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.192822    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv" podUID="4ebb9e23-a65c-4297-9e19-69bc45c4a54f" containerName="registry" containerID="cri-o://65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d" gracePeriod=30
Sep 09 12:21:39.447335 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.447280    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:21:39.536545 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.536505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-bound-sa-token\") pod \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") "
Sep 09 12:21:39.536738 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.536562    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-certificates\") pod \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") "
Sep 09 12:21:39.536738 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.536582    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-trusted-ca\") pod \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") "
Sep 09 12:21:39.536738 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.536609    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-image-registry-private-configuration\") pod \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") "
Sep 09 12:21:39.536738 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.536638    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-ca-trust-extracted\") pod \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") "
Sep 09 12:21:39.536738 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.536719    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-installation-pull-secrets\") pod \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") "
Sep 09 12:21:39.537010 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.536758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls\") pod \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") "
Sep 09 12:21:39.537010 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.536787    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47v8t\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-kube-api-access-47v8t\") pod \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\" (UID: \"4ebb9e23-a65c-4297-9e19-69bc45c4a54f\") "
Sep 09 12:21:39.537111 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.537067    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4ebb9e23-a65c-4297-9e19-69bc45c4a54f" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:21:39.537262 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.537166    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4ebb9e23-a65c-4297-9e19-69bc45c4a54f" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:21:39.539412 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.539371    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4ebb9e23-a65c-4297-9e19-69bc45c4a54f" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:21:39.539538 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.539429    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4ebb9e23-a65c-4297-9e19-69bc45c4a54f" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:21:39.539804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.539777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-kube-api-access-47v8t" (OuterVolumeSpecName: "kube-api-access-47v8t") pod "4ebb9e23-a65c-4297-9e19-69bc45c4a54f" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f"). InnerVolumeSpecName "kube-api-access-47v8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:21:39.539883 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.539811    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4ebb9e23-a65c-4297-9e19-69bc45c4a54f" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:21:39.539883 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.539833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4ebb9e23-a65c-4297-9e19-69bc45c4a54f" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:21:39.545650 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.545620    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4ebb9e23-a65c-4297-9e19-69bc45c4a54f" (UID: "4ebb9e23-a65c-4297-9e19-69bc45c4a54f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:21:39.637975 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.637941    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-installation-pull-secrets\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:21:39.637975 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.637970    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:21:39.637975 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.637979    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47v8t\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-kube-api-access-47v8t\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:21:39.638176 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.637987    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-bound-sa-token\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:21:39.638176 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.637996    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-registry-certificates\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:21:39.638176 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.638003    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-trusted-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:21:39.638176 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.638013    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-image-registry-private-configuration\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:21:39.638176 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.638021    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ebb9e23-a65c-4297-9e19-69bc45c4a54f-ca-trust-extracted\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:21:39.869519 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.869420    2578 generic.go:356] "Generic (PLEG): container finished" podID="ae1623ca-ad35-4f59-a2af-6e19fe017892" containerID="da0faf4bb13e3c58ddf750acb7a283682b9e1fb443d938f0f9ba5bcca15984dc" exitCode=0
Sep 09 12:21:39.869519 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.869500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9" event={"ID":"ae1623ca-ad35-4f59-a2af-6e19fe017892","Type":"ContainerDied","Data":"da0faf4bb13e3c58ddf750acb7a283682b9e1fb443d938f0f9ba5bcca15984dc"}
Sep 09 12:21:39.869913 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.869896    2578 scope.go:117] "RemoveContainer" containerID="da0faf4bb13e3c58ddf750acb7a283682b9e1fb443d938f0f9ba5bcca15984dc"
Sep 09 12:21:39.871142 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.871119    2578 generic.go:356] "Generic (PLEG): container finished" podID="ea6a22dd-7c89-4083-831d-3598aeb4bf58" containerID="6d909fb8fd8119c44ed717fa2fc9abb7af6eed6b5f9447dd0c83a5bd328b68a9" exitCode=0
Sep 09 12:21:39.871277 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.871193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62" event={"ID":"ea6a22dd-7c89-4083-831d-3598aeb4bf58","Type":"ContainerDied","Data":"6d909fb8fd8119c44ed717fa2fc9abb7af6eed6b5f9447dd0c83a5bd328b68a9"}
Sep 09 12:21:39.871545 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.871523    2578 scope.go:117] "RemoveContainer" containerID="6d909fb8fd8119c44ed717fa2fc9abb7af6eed6b5f9447dd0c83a5bd328b68a9"
Sep 09 12:21:39.872602 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.872580    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ebb9e23-a65c-4297-9e19-69bc45c4a54f" containerID="65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d" exitCode=0
Sep 09 12:21:39.872711 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.872649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv" event={"ID":"4ebb9e23-a65c-4297-9e19-69bc45c4a54f","Type":"ContainerDied","Data":"65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d"}
Sep 09 12:21:39.872711 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.872678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv" event={"ID":"4ebb9e23-a65c-4297-9e19-69bc45c4a54f","Type":"ContainerDied","Data":"58b8d38fb99bc7c972879b1e8e722e441fc26b6cffc4b9c702538da5f1d4d5f8"}
Sep 09 12:21:39.872711 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.872696    2578 scope.go:117] "RemoveContainer" containerID="65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d"
Sep 09 12:21:39.872711 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.872692    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4fb58b64-ghqtv"
Sep 09 12:21:39.885169 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.885147    2578 scope.go:117] "RemoveContainer" containerID="65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d"
Sep 09 12:21:39.885811 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:21:39.885678    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d\": container with ID starting with 65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d not found: ID does not exist" containerID="65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d"
Sep 09 12:21:39.885811 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.885704    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d"} err="failed to get container status \"65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d\": rpc error: code = NotFound desc = could not find container \"65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d\": container with ID starting with 65c745e38b18a7f2a6ce6a75c86ecc8b74affc055c1f81e30f8254e8316e8e0d not found: ID does not exist"
Sep 09 12:21:39.900683 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.900656    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c4fb58b64-ghqtv"]
Sep 09 12:21:39.905809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:39.905787    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c4fb58b64-ghqtv"]
Sep 09 12:21:40.188889 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:40.188849    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ebb9e23-a65c-4297-9e19-69bc45c4a54f" path="/var/lib/kubelet/pods/4ebb9e23-a65c-4297-9e19-69bc45c4a54f/volumes"
Sep 09 12:21:40.664105 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:40.664080    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8c9778b-t8sr5_11252896-9746-4dbf-9ca6-01c7d854fab9/router/0.log"
Sep 09 12:21:40.671171 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:40.671149    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-88dzs_d7e96ad5-f31e-45dd-b44e-e49453638364/serve-healthcheck-canary/0.log"
Sep 09 12:21:40.879252 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:40.879190    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-tpw62" event={"ID":"ea6a22dd-7c89-4083-831d-3598aeb4bf58","Type":"ContainerStarted","Data":"2526beca0f486eb8242c181b2fdf07a2121d50b7d45387f281e6156fc181d863"}
Sep 09 12:21:40.881163 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:40.881138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-v5pn9" event={"ID":"ae1623ca-ad35-4f59-a2af-6e19fe017892","Type":"ContainerStarted","Data":"23e6718449d052dd1638e3cca6b50d216d88caadb55f9ab1bcb2a46a1034d856"}
Sep 09 12:21:45.885991 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:45.885956    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:45.892000 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:45.889764    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6c9c7b7756-5smfc"
Sep 09 12:21:49.920194 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:49.920160    2578 generic.go:356] "Generic (PLEG): container finished" podID="c00ec462-3123-4de8-b39f-297d8fa14545" containerID="acb3a34cab60c452f937eb3e8607344ba8570488066a0b786c6deb020ff8aaa4" exitCode=0
Sep 09 12:21:49.920689 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:49.920242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm" event={"ID":"c00ec462-3123-4de8-b39f-297d8fa14545","Type":"ContainerDied","Data":"acb3a34cab60c452f937eb3e8607344ba8570488066a0b786c6deb020ff8aaa4"}
Sep 09 12:21:49.920689 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:49.920641    2578 scope.go:117] "RemoveContainer" containerID="acb3a34cab60c452f937eb3e8607344ba8570488066a0b786c6deb020ff8aaa4"
Sep 09 12:21:50.926779 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:21:50.926744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pgwwm" event={"ID":"c00ec462-3123-4de8-b39f-297d8fa14545","Type":"ContainerStarted","Data":"22867ba8ac9918be64fdf14240f6f0fa3c0e3c2a5058e5dad313f352fb58945c"}
Sep 09 12:22:07.826136 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:07.826094    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:07.842728 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:07.842702    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:08.020053 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:08.020027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:21.556510 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:21.556467    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:22:21.556958 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:21.556754    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="alertmanager" containerID="cri-o://235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c" gracePeriod=120
Sep 09 12:22:21.556958 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:21.556839    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy-web" containerID="cri-o://f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1" gracePeriod=120
Sep 09 12:22:21.556958 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:21.556858    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy" containerID="cri-o://8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25" gracePeriod=120
Sep 09 12:22:21.556958 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:21.556883    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="prom-label-proxy" containerID="cri-o://3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951" gracePeriod=120
Sep 09 12:22:21.556958 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:21.556905    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="config-reloader" containerID="cri-o://9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05" gracePeriod=120
Sep 09 12:22:21.557261 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:21.556837    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy-metric" containerID="cri-o://c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec" gracePeriod=120
Sep 09 12:22:22.077929 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.077898    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerID="3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951" exitCode=0
Sep 09 12:22:22.077929 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.077922    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerID="8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25" exitCode=0
Sep 09 12:22:22.077929 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.077928    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerID="9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05" exitCode=0
Sep 09 12:22:22.077929 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.077933    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerID="235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c" exitCode=0
Sep 09 12:22:22.078192 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.077971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerDied","Data":"3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951"}
Sep 09 12:22:22.078192 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.078061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerDied","Data":"8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25"}
Sep 09 12:22:22.078192 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.078104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerDied","Data":"9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05"}
Sep 09 12:22:22.078192 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.078131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerDied","Data":"235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c"}
Sep 09 12:22:22.803313 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.803287    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:22.933801 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.933763    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-out\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.933979 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.933827    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-main-tls\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.933979 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.933919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-web-config\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.933979 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.933956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-cluster-tls-config\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.933979 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.933977    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.934183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-main-db\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.934183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.934183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934046    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-metrics-client-ca\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.934183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934082    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnlbb\" (UniqueName: \"kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-kube-api-access-xnlbb\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.934183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934123    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-trusted-ca-bundle\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.934183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934163    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-volume\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.934524 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-tls-assets\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.934524 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934269    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\" (UID: \"e2effd5c-94a3-411b-8bd7-ddd419281f7b\") "
Sep 09 12:22:22.934524 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934376    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:22:22.934844 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934559    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-main-db\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:22.934844 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934729    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:22:22.934844 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.934741    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:22:22.937089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.937050    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:22.937664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.937400    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:22.937664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.937419    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-kube-api-access-xnlbb" (OuterVolumeSpecName: "kube-api-access-xnlbb") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "kube-api-access-xnlbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:22:22.937664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.937424    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-volume" (OuterVolumeSpecName: "config-volume") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:22.937664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.937565    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-out" (OuterVolumeSpecName: "config-out") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:22:22.937936 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.937678    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:22.939261 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.939206    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:22.939774 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.939744    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:22:22.941380 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.941357    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:22.948452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:22.948427    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-web-config" (OuterVolumeSpecName: "web-config") pod "e2effd5c-94a3-411b-8bd7-ddd419281f7b" (UID: "e2effd5c-94a3-411b-8bd7-ddd419281f7b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:23.035146 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035119    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnlbb\" (UniqueName: \"kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-kube-api-access-xnlbb\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.035146 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035145    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.035346 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035157    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.035346 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035165    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2effd5c-94a3-411b-8bd7-ddd419281f7b-tls-assets\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.035346 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035173    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.035346 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035181    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2effd5c-94a3-411b-8bd7-ddd419281f7b-config-out\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.035346 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035189    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-main-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.035346 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035236    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-web-config\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.035346 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035246    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-cluster-tls-config\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.035346 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035254    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.035346 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035264    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2effd5c-94a3-411b-8bd7-ddd419281f7b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.035346 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.035272    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2effd5c-94a3-411b-8bd7-ddd419281f7b-metrics-client-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:23.086154 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.086126    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerID="c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec" exitCode=0
Sep 09 12:22:23.086154 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.086148    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerID="f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1" exitCode=0
Sep 09 12:22:23.086391 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.086200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerDied","Data":"c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec"}
Sep 09 12:22:23.086391 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.086240    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.086391 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.086284    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerDied","Data":"f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1"}
Sep 09 12:22:23.086391 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.086312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2effd5c-94a3-411b-8bd7-ddd419281f7b","Type":"ContainerDied","Data":"a60eb5d6df3fc517ce60f4d52c2ce7d6fdfac4bb073425a14ac69043789d201b"}
Sep 09 12:22:23.086391 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.086342    2578 scope.go:117] "RemoveContainer" containerID="3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951"
Sep 09 12:22:23.095068 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.095052    2578 scope.go:117] "RemoveContainer" containerID="c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec"
Sep 09 12:22:23.102430 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.102415    2578 scope.go:117] "RemoveContainer" containerID="8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25"
Sep 09 12:22:23.109596 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.109581    2578 scope.go:117] "RemoveContainer" containerID="f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1"
Sep 09 12:22:23.109663 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.109647    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:22:23.114001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.113978    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:22:23.117370 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.117356    2578 scope.go:117] "RemoveContainer" containerID="9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05"
Sep 09 12:22:23.124304 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.124287    2578 scope.go:117] "RemoveContainer" containerID="235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c"
Sep 09 12:22:23.131332 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.131315    2578 scope.go:117] "RemoveContainer" containerID="f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e"
Sep 09 12:22:23.138533 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.138513    2578 scope.go:117] "RemoveContainer" containerID="3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951"
Sep 09 12:22:23.139007 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:22:23.138875    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951\": container with ID starting with 3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951 not found: ID does not exist" containerID="3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951"
Sep 09 12:22:23.139007 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.138910    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951"} err="failed to get container status \"3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951\": rpc error: code = NotFound desc = could not find container \"3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951\": container with ID starting with 3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951 not found: ID does not exist"
Sep 09 12:22:23.139007 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.138935    2578 scope.go:117] "RemoveContainer" containerID="c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec"
Sep 09 12:22:23.139589 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:22:23.139465    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec\": container with ID starting with c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec not found: ID does not exist" containerID="c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec"
Sep 09 12:22:23.139589 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.139496    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec"} err="failed to get container status \"c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec\": rpc error: code = NotFound desc = could not find container \"c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec\": container with ID starting with c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec not found: ID does not exist"
Sep 09 12:22:23.139589 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.139518    2578 scope.go:117] "RemoveContainer" containerID="8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25"
Sep 09 12:22:23.140023 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:22:23.139998    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25\": container with ID starting with 8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25 not found: ID does not exist" containerID="8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25"
Sep 09 12:22:23.140120 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140031    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25"} err="failed to get container status \"8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25\": rpc error: code = NotFound desc = could not find container \"8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25\": container with ID starting with 8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25 not found: ID does not exist"
Sep 09 12:22:23.140120 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140051    2578 scope.go:117] "RemoveContainer" containerID="f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1"
Sep 09 12:22:23.140375 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:22:23.140354    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1\": container with ID starting with f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1 not found: ID does not exist" containerID="f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1"
Sep 09 12:22:23.140375 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140377    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1"} err="failed to get container status \"f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1\": rpc error: code = NotFound desc = could not find container \"f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1\": container with ID starting with f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1 not found: ID does not exist"
Sep 09 12:22:23.140498 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140394    2578 scope.go:117] "RemoveContainer" containerID="9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05"
Sep 09 12:22:23.140498 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140474    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:22:23.140668 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:22:23.140651    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05\": container with ID starting with 9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05 not found: ID does not exist" containerID="9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05"
Sep 09 12:22:23.140709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140673    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05"} err="failed to get container status \"9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05\": rpc error: code = NotFound desc = could not find container \"9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05\": container with ID starting with 9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05 not found: ID does not exist"
Sep 09 12:22:23.140709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140687    2578 scope.go:117] "RemoveContainer" containerID="235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c"
Sep 09 12:22:23.140841 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140825    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy-web"
Sep 09 12:22:23.140877 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140844    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy-web"
Sep 09 12:22:23.140877 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140855    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="alertmanager"
Sep 09 12:22:23.140877 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140860    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="alertmanager"
Sep 09 12:22:23.140877 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140866    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy"
Sep 09 12:22:23.140877 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140871    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140878    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="config-reloader"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140886    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="config-reloader"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:22:23.140894    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c\": container with ID starting with 235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c not found: ID does not exist" containerID="235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140903    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ebb9e23-a65c-4297-9e19-69bc45c4a54f" containerName="registry"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140910    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ebb9e23-a65c-4297-9e19-69bc45c4a54f" containerName="registry"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140908    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c"} err="failed to get container status \"235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c\": rpc error: code = NotFound desc = could not find container \"235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c\": container with ID starting with 235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c not found: ID does not exist"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140919    2578 scope.go:117] "RemoveContainer" containerID="f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140922    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="init-config-reloader"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140928    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="init-config-reloader"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140934    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy-metric"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140942    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy-metric"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140955    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="prom-label-proxy"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.140960    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="prom-label-proxy"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141012    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="prom-label-proxy"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141024    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy"
Sep 09 12:22:23.141030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141034    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="config-reloader"
Sep 09 12:22:23.141613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141045    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy-metric"
Sep 09 12:22:23.141613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141052    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ebb9e23-a65c-4297-9e19-69bc45c4a54f" containerName="registry"
Sep 09 12:22:23.141613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141058    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="kube-rbac-proxy-web"
Sep 09 12:22:23.141613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141064    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" containerName="alertmanager"
Sep 09 12:22:23.141613 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:22:23.141144    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e\": container with ID starting with f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e not found: ID does not exist" containerID="f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e"
Sep 09 12:22:23.141613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141161    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e"} err="failed to get container status \"f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e\": rpc error: code = NotFound desc = could not find container \"f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e\": container with ID starting with f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e not found: ID does not exist"
Sep 09 12:22:23.141613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141174    2578 scope.go:117] "RemoveContainer" containerID="3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951"
Sep 09 12:22:23.141613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141434    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951"} err="failed to get container status \"3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951\": rpc error: code = NotFound desc = could not find container \"3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951\": container with ID starting with 3f8b51fa63d2db7ee7e2ab67ce1af14773baf5b2a42f824987cf550610344951 not found: ID does not exist"
Sep 09 12:22:23.141613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141450    2578 scope.go:117] "RemoveContainer" containerID="c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec"
Sep 09 12:22:23.141934 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141767    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec"} err="failed to get container status \"c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec\": rpc error: code = NotFound desc = could not find container \"c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec\": container with ID starting with c664b6c61ac0b8eb5b35764e0eb64070cc4109b959d53cb5c9457bc6ab29bdec not found: ID does not exist"
Sep 09 12:22:23.141934 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.141788    2578 scope.go:117] "RemoveContainer" containerID="8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25"
Sep 09 12:22:23.142060 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.142040    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25"} err="failed to get container status \"8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25\": rpc error: code = NotFound desc = could not find container \"8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25\": container with ID starting with 8620b44e0378be49f691fb3941171268aa07c24602c137302b6c502392524c25 not found: ID does not exist"
Sep 09 12:22:23.142115 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.142060    2578 scope.go:117] "RemoveContainer" containerID="f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1"
Sep 09 12:22:23.142331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.142312    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1"} err="failed to get container status \"f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1\": rpc error: code = NotFound desc = could not find container \"f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1\": container with ID starting with f941ba8a74761595f978a0d7d8dcaf6948f525da2aa356b44ac17f95ab7fe7e1 not found: ID does not exist"
Sep 09 12:22:23.142331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.142330    2578 scope.go:117] "RemoveContainer" containerID="9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05"
Sep 09 12:22:23.142626 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.142603    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05"} err="failed to get container status \"9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05\": rpc error: code = NotFound desc = could not find container \"9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05\": container with ID starting with 9b98a36485171d3fa9afff620b3363ea1813c8bef9bde0ac04ed76a5ba852b05 not found: ID does not exist"
Sep 09 12:22:23.142690 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.142628    2578 scope.go:117] "RemoveContainer" containerID="235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c"
Sep 09 12:22:23.142893 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.142875    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c"} err="failed to get container status \"235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c\": rpc error: code = NotFound desc = could not find container \"235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c\": container with ID starting with 235fee80454196c45e5c617072f098665c2a0b26410c9151af166b5c621b0a1c not found: ID does not exist"
Sep 09 12:22:23.142949 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.142894    2578 scope.go:117] "RemoveContainer" containerID="f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e"
Sep 09 12:22:23.143197 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.143180    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e"} err="failed to get container status \"f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e\": rpc error: code = NotFound desc = could not find container \"f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e\": container with ID starting with f9fafc8e04b87c8d4576b59f1b124b29395f0df4cf62ec4e385eb8034d2fb52e not found: ID does not exist"
Sep 09 12:22:23.144663 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.144649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.146571 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.146546    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 12:22:23.146571 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.146546    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 12:22:23.146721 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.146571    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 12:22:23.147070 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.146826    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 12:22:23.147070 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.146830    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 12:22:23.147070 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.146894    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 12:22:23.147070 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.146851    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-24vfz\""
Sep 09 12:22:23.147070 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.146941    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 12:22:23.147070 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.146830    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 12:22:23.151632 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.151611    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 12:22:23.161928 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.161902    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:22:23.237688 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a290f195-8dc2-418c-8b43-67515c1e4eed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.237688 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a290f195-8dc2-418c-8b43-67515c1e4eed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.237688 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a290f195-8dc2-418c-8b43-67515c1e4eed-config-out\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.237688 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.237948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.237948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a290f195-8dc2-418c-8b43-67515c1e4eed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.237948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.237948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.237948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a290f195-8dc2-418c-8b43-67515c1e4eed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.237948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46jc\" (UniqueName: \"kubernetes.io/projected/a290f195-8dc2-418c-8b43-67515c1e4eed-kube-api-access-z46jc\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.237948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-web-config\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.238247 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.237983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-config-volume\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.238247 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.238027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.338997 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.338946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339190 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a290f195-8dc2-418c-8b43-67515c1e4eed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339190 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339190 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339190 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a290f195-8dc2-418c-8b43-67515c1e4eed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339190 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46jc\" (UniqueName: \"kubernetes.io/projected/a290f195-8dc2-418c-8b43-67515c1e4eed-kube-api-access-z46jc\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339190 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-web-config\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339190 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-config-volume\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339615 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339615 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a290f195-8dc2-418c-8b43-67515c1e4eed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339615 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a290f195-8dc2-418c-8b43-67515c1e4eed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339615 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a290f195-8dc2-418c-8b43-67515c1e4eed-config-out\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.339615 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.339401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.340319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.340288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a290f195-8dc2-418c-8b43-67515c1e4eed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.340631 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.340602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a290f195-8dc2-418c-8b43-67515c1e4eed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.341229 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.341185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a290f195-8dc2-418c-8b43-67515c1e4eed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.342374 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.342188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a290f195-8dc2-418c-8b43-67515c1e4eed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.342374 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.342310    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.342534 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.342431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.342598 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.342528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.342598 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.342575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.342858 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.342840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-web-config\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.342858 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.342851    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.343336 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.343314    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a290f195-8dc2-418c-8b43-67515c1e4eed-config-out\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.344260 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.344244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a290f195-8dc2-418c-8b43-67515c1e4eed-config-volume\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.347574 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.347557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46jc\" (UniqueName: \"kubernetes.io/projected/a290f195-8dc2-418c-8b43-67515c1e4eed-kube-api-access-z46jc\") pod \"alertmanager-main-0\" (UID: \"a290f195-8dc2-418c-8b43-67515c1e4eed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.454898 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.454862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 12:22:23.739318 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:23.739285    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 12:22:23.742639 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:22:23.742609    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda290f195_8dc2_418c_8b43_67515c1e4eed.slice/crio-287fbfaf0a549483d5c4530477d6ffc828a785f3c761c43a19491ae5c7d7d2f4 WatchSource:0}: Error finding container 287fbfaf0a549483d5c4530477d6ffc828a785f3c761c43a19491ae5c7d7d2f4: Status 404 returned error can't find the container with id 287fbfaf0a549483d5c4530477d6ffc828a785f3c761c43a19491ae5c7d7d2f4
Sep 09 12:22:24.092001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:24.091905    2578 generic.go:356] "Generic (PLEG): container finished" podID="a290f195-8dc2-418c-8b43-67515c1e4eed" containerID="3adf2608e824a1e8ef9112649e0b8b5aae2e2f858375ab0a81e41380ecb7180e" exitCode=0
Sep 09 12:22:24.092001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:24.091990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a290f195-8dc2-418c-8b43-67515c1e4eed","Type":"ContainerDied","Data":"3adf2608e824a1e8ef9112649e0b8b5aae2e2f858375ab0a81e41380ecb7180e"}
Sep 09 12:22:24.092528 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:24.092060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a290f195-8dc2-418c-8b43-67515c1e4eed","Type":"ContainerStarted","Data":"287fbfaf0a549483d5c4530477d6ffc828a785f3c761c43a19491ae5c7d7d2f4"}
Sep 09 12:22:24.186582 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:24.186553    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2effd5c-94a3-411b-8bd7-ddd419281f7b" path="/var/lib/kubelet/pods/e2effd5c-94a3-411b-8bd7-ddd419281f7b/volumes"
Sep 09 12:22:25.100928 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.100898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a290f195-8dc2-418c-8b43-67515c1e4eed","Type":"ContainerStarted","Data":"b6507c3d3405bed893899aa2f0b1a36d44fa2b8b29018dff9a220fa89daca90d"}
Sep 09 12:22:25.101352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.100959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a290f195-8dc2-418c-8b43-67515c1e4eed","Type":"ContainerStarted","Data":"8ce6578f3d165747a42f3a96c513ab0707f9e0bf96c57008d08c61443cd1c298"}
Sep 09 12:22:25.101352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.100984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a290f195-8dc2-418c-8b43-67515c1e4eed","Type":"ContainerStarted","Data":"89f0a88b64232cec69d6815adb9cf86b85595683f540be39f106c07f413b879d"}
Sep 09 12:22:25.101352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.101008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a290f195-8dc2-418c-8b43-67515c1e4eed","Type":"ContainerStarted","Data":"9333771e899a9ba6a096a9896ccac4e90ab1404fdf3af5ed860c5ffb01753667"}
Sep 09 12:22:25.101352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.101032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a290f195-8dc2-418c-8b43-67515c1e4eed","Type":"ContainerStarted","Data":"0aab5b1bcac3859272dd15e12197c5688d001c5b7d145b5a418f08b67e902f9a"}
Sep 09 12:22:25.101352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.101056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a290f195-8dc2-418c-8b43-67515c1e4eed","Type":"ContainerStarted","Data":"1441686316e7df2e33a072c9c832f8ff3b624a1bfe6f9aea57a0420d58d1d092"}
Sep 09 12:22:25.129436 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.129389    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.113540228 podStartE2EDuration="2.129364345s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.824117ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:22:25.128473509 +0000 UTC m=+165.472071470" watchObservedRunningTime="2026-09-09 12:22:25.129364345 +0000 UTC m=+165.472962294"
Sep 09 12:22:25.904586 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.904549    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:22:25.904882 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.904839    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy" containerID="cri-o://65b59f2fd8396d2c4557e9c14c2bed3f2ae8ed3e8dceddc62f7648064e28c6c5" gracePeriod=600
Sep 09 12:22:25.905020 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.904876    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1fc6aef40d49ffe2368a7f9d1cb5315d6644a783ef57d945fbabcec6d486cd32" gracePeriod=600
Sep 09 12:22:25.905020 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.904890    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="thanos-sidecar" containerID="cri-o://642b8a813c3b377bfdbedae57b66581865b10bbd2689d564e42650b2200ac740" gracePeriod=600
Sep 09 12:22:25.905020 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.904901    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy-web" containerID="cri-o://6e9d6e5f19126d49f1c6f038f0ec749b212c7f095c6bda6d04a52b994b28952a" gracePeriod=600
Sep 09 12:22:25.905020 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.904830    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="prometheus" containerID="cri-o://9d4cdf40a97c7572ec40ce580efa2a27d89f8d1ae17344e59a359d13a21bc34d" gracePeriod=600
Sep 09 12:22:25.905020 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:25.904882    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="config-reloader" containerID="cri-o://425efd287744b767f825ceaae0d81826aec51bc724f4f540f35749604eec7c4f" gracePeriod=600
Sep 09 12:22:26.109823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.109794    2578 generic.go:356] "Generic (PLEG): container finished" podID="99b49a95-daf0-4522-b783-a259fa6213fb" containerID="1fc6aef40d49ffe2368a7f9d1cb5315d6644a783ef57d945fbabcec6d486cd32" exitCode=0
Sep 09 12:22:26.109823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.109819    2578 generic.go:356] "Generic (PLEG): container finished" podID="99b49a95-daf0-4522-b783-a259fa6213fb" containerID="65b59f2fd8396d2c4557e9c14c2bed3f2ae8ed3e8dceddc62f7648064e28c6c5" exitCode=0
Sep 09 12:22:26.109823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.109825    2578 generic.go:356] "Generic (PLEG): container finished" podID="99b49a95-daf0-4522-b783-a259fa6213fb" containerID="6e9d6e5f19126d49f1c6f038f0ec749b212c7f095c6bda6d04a52b994b28952a" exitCode=0
Sep 09 12:22:26.109823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.109831    2578 generic.go:356] "Generic (PLEG): container finished" podID="99b49a95-daf0-4522-b783-a259fa6213fb" containerID="642b8a813c3b377bfdbedae57b66581865b10bbd2689d564e42650b2200ac740" exitCode=0
Sep 09 12:22:26.110335 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.109837    2578 generic.go:356] "Generic (PLEG): container finished" podID="99b49a95-daf0-4522-b783-a259fa6213fb" containerID="425efd287744b767f825ceaae0d81826aec51bc724f4f540f35749604eec7c4f" exitCode=0
Sep 09 12:22:26.110335 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.109843    2578 generic.go:356] "Generic (PLEG): container finished" podID="99b49a95-daf0-4522-b783-a259fa6213fb" containerID="9d4cdf40a97c7572ec40ce580efa2a27d89f8d1ae17344e59a359d13a21bc34d" exitCode=0
Sep 09 12:22:26.110335 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.109874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerDied","Data":"1fc6aef40d49ffe2368a7f9d1cb5315d6644a783ef57d945fbabcec6d486cd32"}
Sep 09 12:22:26.110335 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.109954    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerDied","Data":"65b59f2fd8396d2c4557e9c14c2bed3f2ae8ed3e8dceddc62f7648064e28c6c5"}
Sep 09 12:22:26.110335 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.109997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerDied","Data":"6e9d6e5f19126d49f1c6f038f0ec749b212c7f095c6bda6d04a52b994b28952a"}
Sep 09 12:22:26.110335 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.110040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerDied","Data":"642b8a813c3b377bfdbedae57b66581865b10bbd2689d564e42650b2200ac740"}
Sep 09 12:22:26.110335 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.110079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerDied","Data":"425efd287744b767f825ceaae0d81826aec51bc724f4f540f35749604eec7c4f"}
Sep 09 12:22:26.110335 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.110108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerDied","Data":"9d4cdf40a97c7572ec40ce580efa2a27d89f8d1ae17344e59a359d13a21bc34d"}
Sep 09 12:22:26.145139 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.145116    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:26.267279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267186    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-metrics-client-certs\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267246    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-thanos-prometheus-http-client-file\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267276    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-config\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267373    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-0\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267405    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-kubelet-serving-ca-bundle\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267460    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-serving-certs-ca-bundle\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267488    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267521    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-web-config\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267784 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267562    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-1\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267784 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267601    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-db\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267784 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx6w7\" (UniqueName: \"kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-kube-api-access-qx6w7\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267784 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267663    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267784 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267706    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-kube-rbac-proxy\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267784 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267736    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-tls-assets\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.267784 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267763    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-metrics-client-ca\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.268097 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267801    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-config-out\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.268097 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267825    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-trusted-ca-bundle\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.268097 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-tls\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.268097 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267925    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-2\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.268097 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267949    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-grpc-tls\") pod \"99b49a95-daf0-4522-b783-a259fa6213fb\" (UID: \"99b49a95-daf0-4522-b783-a259fa6213fb\") "
Sep 09 12:22:26.268097 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.267799    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:22:26.268411 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.268144    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:22:26.268462 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.268442    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.268508 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.268461    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.268726 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.268621    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:22:26.268941 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.268871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:22:26.269328 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.269285    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:22:26.270272 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.270246    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:22:26.270653 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.270619    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-config" (OuterVolumeSpecName: "config") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:26.270653 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.270638    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:26.270798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.270690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:26.270974 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.270921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:26.270974 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.270668    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:22:26.271247 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.270989    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:22:26.271943 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.271917    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:26.272435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.272409    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:26.272757 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.272719    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:26.272948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.272929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-kube-api-access-qx6w7" (OuterVolumeSpecName: "kube-api-access-qx6w7") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "kube-api-access-qx6w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:22:26.273036 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.273018    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:22:26.273207 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.273191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:26.273432 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.273415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-config-out" (OuterVolumeSpecName: "config-out") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:22:26.283383 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.283362    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-web-config" (OuterVolumeSpecName: "web-config") pod "99b49a95-daf0-4522-b783-a259fa6213fb" (UID: "99b49a95-daf0-4522-b783-a259fa6213fb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:22:26.368896 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368863    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.368896 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368892    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.368896 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368904    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-web-config\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368915    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-db\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368924    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx6w7\" (UniqueName: \"kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-kube-api-access-qx6w7\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368933    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368941    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-kube-rbac-proxy\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368948    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99b49a95-daf0-4522-b783-a259fa6213fb-tls-assets\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368956    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-configmap-metrics-client-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368963    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99b49a95-daf0-4522-b783-a259fa6213fb-config-out\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368971    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368979    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368987    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.368994    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-grpc-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.369002    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-secret-metrics-client-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.369009    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.369017    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/99b49a95-daf0-4522-b783-a259fa6213fb-config\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:26.369121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:26.369025    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/99b49a95-daf0-4522-b783-a259fa6213fb-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:22:27.117114 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.117083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"99b49a95-daf0-4522-b783-a259fa6213fb","Type":"ContainerDied","Data":"b01f34ecdcf739ddc67ef27ca24da7a8faa2476d2cc6f2cee8b0f7b4c9e80c94"}
Sep 09 12:22:27.117527 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.117160    2578 scope.go:117] "RemoveContainer" containerID="1fc6aef40d49ffe2368a7f9d1cb5315d6644a783ef57d945fbabcec6d486cd32"
Sep 09 12:22:27.117527 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.117159    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.132159 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.132135    2578 scope.go:117] "RemoveContainer" containerID="65b59f2fd8396d2c4557e9c14c2bed3f2ae8ed3e8dceddc62f7648064e28c6c5"
Sep 09 12:22:27.139816 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.139788    2578 scope.go:117] "RemoveContainer" containerID="6e9d6e5f19126d49f1c6f038f0ec749b212c7f095c6bda6d04a52b994b28952a"
Sep 09 12:22:27.146776 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.146753    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:22:27.147396 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.147369    2578 scope.go:117] "RemoveContainer" containerID="642b8a813c3b377bfdbedae57b66581865b10bbd2689d564e42650b2200ac740"
Sep 09 12:22:27.151953 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.151934    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:22:27.154728 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.154716    2578 scope.go:117] "RemoveContainer" containerID="425efd287744b767f825ceaae0d81826aec51bc724f4f540f35749604eec7c4f"
Sep 09 12:22:27.161750 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.161721    2578 scope.go:117] "RemoveContainer" containerID="9d4cdf40a97c7572ec40ce580efa2a27d89f8d1ae17344e59a359d13a21bc34d"
Sep 09 12:22:27.169071 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.169054    2578 scope.go:117] "RemoveContainer" containerID="2cc9b80db7c8a031750758966a87cfc437ef2076991466446ab6e9637d779aee"
Sep 09 12:22:27.177639 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.177618    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:22:27.177994 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.177978    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy-thanos"
Sep 09 12:22:27.178062 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.177997    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy-thanos"
Sep 09 12:22:27.178062 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178012    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="init-config-reloader"
Sep 09 12:22:27.178062 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178020    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="init-config-reloader"
Sep 09 12:22:27.178062 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178043    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy"
Sep 09 12:22:27.178062 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178053    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178063    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="prometheus"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178073    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="prometheus"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178083    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy-web"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178092    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy-web"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178106    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="thanos-sidecar"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178114    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="thanos-sidecar"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178127    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="config-reloader"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178135    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="config-reloader"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178248    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy-thanos"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178262    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="config-reloader"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178273    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="thanos-sidecar"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178283    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="prometheus"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178292    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy"
Sep 09 12:22:27.178323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.178303    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" containerName="kube-rbac-proxy-web"
Sep 09 12:22:27.182438 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.182421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.184756 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.184723    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 12:22:27.184756 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.184728    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 12:22:27.184919 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.184769    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 12:22:27.184919 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.184811    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9aefhn32l0u95\""
Sep 09 12:22:27.184919 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.184867    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 12:22:27.185065 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.185006    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 12:22:27.185117 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.185065    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 12:22:27.185117 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.185081    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 12:22:27.185259 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.185201    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 12:22:27.185375 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.185358    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 12:22:27.185444 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.185380    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 12:22:27.185444 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.185421    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 12:22:27.185521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.185489    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 12:22:27.185570 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.185530    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cx6vv\""
Sep 09 12:22:27.187589 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.187572    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 12:22:27.190089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.190074    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 12:22:27.196507 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.196485    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:22:27.276603 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276603 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276811 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276811 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276811 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lprwm\" (UniqueName: \"kubernetes.io/projected/023c9c13-178a-43e2-bdc8-11ff23633d15-kube-api-access-lprwm\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276811 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276811 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-config\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276811 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/023c9c13-178a-43e2-bdc8-11ff23633d15-config-out\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276988 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276988 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276988 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276988 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276988 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-web-config\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.276988 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.277155 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.276992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.277155 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.277015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.277155 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.277033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.277155 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.277061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.277155 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.277099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.277155 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.277135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/023c9c13-178a-43e2-bdc8-11ff23633d15-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378532 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378532 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-config\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378684 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/023c9c13-178a-43e2-bdc8-11ff23633d15-config-out\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378684 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378684 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378684 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378839 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378839 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-web-config\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378839 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378839 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378839 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.378839 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.379129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.379129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.379129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.378976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/023c9c13-178a-43e2-bdc8-11ff23633d15-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.379129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.379012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.379129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.379047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.379129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.379068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.379129 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.379106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.379505 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.379142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lprwm\" (UniqueName: \"kubernetes.io/projected/023c9c13-178a-43e2-bdc8-11ff23633d15-kube-api-access-lprwm\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.379505 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.379359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.379747 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.379715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.379860 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.379834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.380287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.380265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.380412 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.380387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.381755 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.381728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/023c9c13-178a-43e2-bdc8-11ff23633d15-config-out\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.382547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.382165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-config\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.382547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.382277    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/023c9c13-178a-43e2-bdc8-11ff23633d15-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.382686 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.382652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.382930 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.382905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.383520 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.383403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.383520 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.383462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.383676 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.383608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.383676 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.383638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-web-config\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.383987 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.383968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/023c9c13-178a-43e2-bdc8-11ff23633d15-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.384319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.384301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.384735 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.384717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.385382 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.385363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.385754 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.385737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/023c9c13-178a-43e2-bdc8-11ff23633d15-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.387735 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.387711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lprwm\" (UniqueName: \"kubernetes.io/projected/023c9c13-178a-43e2-bdc8-11ff23633d15-kube-api-access-lprwm\") pod \"prometheus-k8s-0\" (UID: \"023c9c13-178a-43e2-bdc8-11ff23633d15\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.492706 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.492617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:22:27.771827 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:27.771801    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 12:22:27.773652 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:22:27.773623    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023c9c13_178a_43e2_bdc8_11ff23633d15.slice/crio-1f89af086ecd001d533d3da878d1b7a41caaa5acb603528cad6fa0ca9a1f51db WatchSource:0}: Error finding container 1f89af086ecd001d533d3da878d1b7a41caaa5acb603528cad6fa0ca9a1f51db: Status 404 returned error can't find the container with id 1f89af086ecd001d533d3da878d1b7a41caaa5acb603528cad6fa0ca9a1f51db
Sep 09 12:22:28.123667 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:28.123580    2578 generic.go:356] "Generic (PLEG): container finished" podID="023c9c13-178a-43e2-bdc8-11ff23633d15" containerID="b37890e4108af572d3120eaf39221caa88902794809d8a8ed4aa68b806b47333" exitCode=0
Sep 09 12:22:28.123667 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:28.123648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"023c9c13-178a-43e2-bdc8-11ff23633d15","Type":"ContainerDied","Data":"b37890e4108af572d3120eaf39221caa88902794809d8a8ed4aa68b806b47333"}
Sep 09 12:22:28.124048 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:28.123706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"023c9c13-178a-43e2-bdc8-11ff23633d15","Type":"ContainerStarted","Data":"1f89af086ecd001d533d3da878d1b7a41caaa5acb603528cad6fa0ca9a1f51db"}
Sep 09 12:22:28.186594 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:28.186567    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b49a95-daf0-4522-b783-a259fa6213fb" path="/var/lib/kubelet/pods/99b49a95-daf0-4522-b783-a259fa6213fb/volumes"
Sep 09 12:22:29.131696 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:29.131656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"023c9c13-178a-43e2-bdc8-11ff23633d15","Type":"ContainerStarted","Data":"314436fa6b9772a30860e928f2af56a3a75cbd95ab27a14e0a8311836dbaa606"}
Sep 09 12:22:29.132082 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:29.131773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"023c9c13-178a-43e2-bdc8-11ff23633d15","Type":"ContainerStarted","Data":"b458d0483fc6843eebb85de5f8197ab5506b7aa11d9fc73095a1b74300b1cb15"}
Sep 09 12:22:29.132082 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:29.131821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"023c9c13-178a-43e2-bdc8-11ff23633d15","Type":"ContainerStarted","Data":"e897f869724884513f301dc6887a9214718f7e58ba1bce3b412442da434865c1"}
Sep 09 12:22:29.132082 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:29.131856    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"023c9c13-178a-43e2-bdc8-11ff23633d15","Type":"ContainerStarted","Data":"57249af6b709bcbfb4a9807c2e507c382c7bfc2d44ad8656e653c78d8969b8af"}
Sep 09 12:22:29.132082 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:29.131891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"023c9c13-178a-43e2-bdc8-11ff23633d15","Type":"ContainerStarted","Data":"010ffbb6f265b1d53fa33dbfb169e2c563f76b2be220f83c126c0593008c6fda"}
Sep 09 12:22:29.132082 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:29.131933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"023c9c13-178a-43e2-bdc8-11ff23633d15","Type":"ContainerStarted","Data":"95835f0d442e2d227cad7e40c8460b1d81dd37141bd2e87dfa0d95e3827c3391"}
Sep 09 12:22:29.163641 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:29.163580    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.140794981 podStartE2EDuration="2.163561426s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.766445ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:22:29.162202875 +0000 UTC m=+169.505800824" watchObservedRunningTime="2026-09-09 12:22:29.163561426 +0000 UTC m=+169.507159376"
Sep 09 12:22:32.493041 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:22:32.492945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:23:27.493418 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:23:27.493379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:23:27.509665 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:23:27.509641    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:23:28.390496 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:23:28.390468    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 12:24:40.082798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:24:40.082768    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:24:40.087914 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:24:40.087893    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:24:40.094594 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:24:40.094564    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 12:25:48.087568 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.087488    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"]
Sep 09 12:25:48.090644 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.090628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.093661 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.093635    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 12:25:48.093661 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.093654    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:25:48.093661 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.093640    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5rjzx\""
Sep 09 12:25:48.093903 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.093641    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 12:25:48.093903 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.093641    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 12:25:48.093903 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.093688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 12:25:48.099333 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.099312    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"]
Sep 09 12:25:48.264280 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.264242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0d9a27ad-aebe-4c40-a579-a140aea302ac-manager-config\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.264464 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.264353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d9a27ad-aebe-4c40-a579-a140aea302ac-cert\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.264464 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.264384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6jcm\" (UniqueName: \"kubernetes.io/projected/0d9a27ad-aebe-4c40-a579-a140aea302ac-kube-api-access-j6jcm\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.264464 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.264442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d9a27ad-aebe-4c40-a579-a140aea302ac-metrics-cert\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.365093 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.364998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d9a27ad-aebe-4c40-a579-a140aea302ac-cert\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.365093 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.365048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6jcm\" (UniqueName: \"kubernetes.io/projected/0d9a27ad-aebe-4c40-a579-a140aea302ac-kube-api-access-j6jcm\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.365330 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.365115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d9a27ad-aebe-4c40-a579-a140aea302ac-metrics-cert\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.365330 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.365162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0d9a27ad-aebe-4c40-a579-a140aea302ac-manager-config\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.365875 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.365844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0d9a27ad-aebe-4c40-a579-a140aea302ac-manager-config\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.367731 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.367706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d9a27ad-aebe-4c40-a579-a140aea302ac-cert\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.368088 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.368066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d9a27ad-aebe-4c40-a579-a140aea302ac-metrics-cert\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.373075 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.373050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6jcm\" (UniqueName: \"kubernetes.io/projected/0d9a27ad-aebe-4c40-a579-a140aea302ac-kube-api-access-j6jcm\") pod \"lws-controller-manager-5465bf697c-9m8xc\" (UID: \"0d9a27ad-aebe-4c40-a579-a140aea302ac\") " pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.400458 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.400432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:48.668360 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.668341    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"]
Sep 09 12:25:48.670656 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:25:48.670627    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9a27ad_aebe_4c40_a579_a140aea302ac.slice/crio-b68e9e61ac7bfe71477fc5d1615c828233e3197e7e17a445080a780a2ceef5fc WatchSource:0}: Error finding container b68e9e61ac7bfe71477fc5d1615c828233e3197e7e17a445080a780a2ceef5fc: Status 404 returned error can't find the container with id b68e9e61ac7bfe71477fc5d1615c828233e3197e7e17a445080a780a2ceef5fc
Sep 09 12:25:48.672674 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.672659    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:25:48.966680 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:48.966598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc" event={"ID":"0d9a27ad-aebe-4c40-a579-a140aea302ac","Type":"ContainerStarted","Data":"b68e9e61ac7bfe71477fc5d1615c828233e3197e7e17a445080a780a2ceef5fc"}
Sep 09 12:25:51.982320 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:51.982283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc" event={"ID":"0d9a27ad-aebe-4c40-a579-a140aea302ac","Type":"ContainerStarted","Data":"f33d3a65e7e893ee00cd3d6f71dc57761a316a9a6c38f7bf6e1f4f5aab6f0357"}
Sep 09 12:25:51.982728 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:51.982446    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:25:52.002604 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:25:52.002559    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc" podStartSLOduration=0.807961905 podStartE2EDuration="4.002541998s" totalImagesPullingTime="3.194580093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:25:52.001004241 +0000 UTC m=+372.344602188" watchObservedRunningTime="2026-09-09 12:25:52.002541998 +0000 UTC m=+372.346139945"
Sep 09 12:26:02.988883 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:02.988854    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5465bf697c-9m8xc"
Sep 09 12:26:03.599974 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.599913    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"]
Sep 09 12:26:03.603260 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.603243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.605111 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.605089    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 12:26:03.605268 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.605250    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 12:26:03.605343 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.605256    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 12:26:03.605401 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.605392    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4l7r7\""
Sep 09 12:26:03.605454 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.605425    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 12:26:03.622142 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.622114    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"]
Sep 09 12:26:03.694895 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.694855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsszr\" (UniqueName: \"kubernetes.io/projected/d2536917-17ea-419b-84ee-a01daefd69e7-kube-api-access-nsszr\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.694895 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.694901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2536917-17ea-419b-84ee-a01daefd69e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.695106 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.694952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2536917-17ea-419b-84ee-a01daefd69e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.695106 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.695034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2536917-17ea-419b-84ee-a01daefd69e7-tmp\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.695106 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.695073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2536917-17ea-419b-84ee-a01daefd69e7-manifests\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.796505 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.796474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2536917-17ea-419b-84ee-a01daefd69e7-tmp\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.796664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.796514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2536917-17ea-419b-84ee-a01daefd69e7-manifests\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.796664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.796544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsszr\" (UniqueName: \"kubernetes.io/projected/d2536917-17ea-419b-84ee-a01daefd69e7-kube-api-access-nsszr\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.796664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.796563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2536917-17ea-419b-84ee-a01daefd69e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.796664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.796603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2536917-17ea-419b-84ee-a01daefd69e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.796990 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.796963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2536917-17ea-419b-84ee-a01daefd69e7-manifests\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.799080 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.799050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2536917-17ea-419b-84ee-a01daefd69e7-tmp\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.799175 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.799141    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2536917-17ea-419b-84ee-a01daefd69e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.799175 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.799155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2536917-17ea-419b-84ee-a01daefd69e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.807919 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.807899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsszr\" (UniqueName: \"kubernetes.io/projected/d2536917-17ea-419b-84ee-a01daefd69e7-kube-api-access-nsszr\") pod \"opendatahub-operator-controller-manager-6c5d49f968-hfkz2\" (UID: \"d2536917-17ea-419b-84ee-a01daefd69e7\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:03.913643 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:03.913613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:04.198546 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:04.198523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"]
Sep 09 12:26:04.202138 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:26:04.202110    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2536917_17ea_419b_84ee_a01daefd69e7.slice/crio-91047854144b6f71b2a38e046ac7fb15e3fc03a65e8b83620fa3498a8e955a06 WatchSource:0}: Error finding container 91047854144b6f71b2a38e046ac7fb15e3fc03a65e8b83620fa3498a8e955a06: Status 404 returned error can't find the container with id 91047854144b6f71b2a38e046ac7fb15e3fc03a65e8b83620fa3498a8e955a06
Sep 09 12:26:05.047235 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:05.047142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2" event={"ID":"d2536917-17ea-419b-84ee-a01daefd69e7","Type":"ContainerStarted","Data":"91047854144b6f71b2a38e046ac7fb15e3fc03a65e8b83620fa3498a8e955a06"}
Sep 09 12:26:08.061993 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:08.061957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2" event={"ID":"d2536917-17ea-419b-84ee-a01daefd69e7","Type":"ContainerStarted","Data":"24fcd3d5057640789a3035e1b7d3f459b88f98a7d2d0ea7562b893ea175620f5"}
Sep 09 12:26:09.067478 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:09.067439    2578 generic.go:356] "Generic (PLEG): container finished" podID="d2536917-17ea-419b-84ee-a01daefd69e7" containerID="24fcd3d5057640789a3035e1b7d3f459b88f98a7d2d0ea7562b893ea175620f5" exitCode=0
Sep 09 12:26:09.067860 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:09.067489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2" event={"ID":"d2536917-17ea-419b-84ee-a01daefd69e7","Type":"ContainerDied","Data":"24fcd3d5057640789a3035e1b7d3f459b88f98a7d2d0ea7562b893ea175620f5"}
Sep 09 12:26:10.074494 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:10.074462    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2" event={"ID":"d2536917-17ea-419b-84ee-a01daefd69e7","Type":"ContainerStarted","Data":"54590bc9fd70805f28ad84e5c2eed74eb76b0a5c4659617c648a7aeb9f28d923"}
Sep 09 12:26:10.074869 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:10.074594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:10.099264 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:10.099202    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2" podStartSLOduration=2.631397415 podStartE2EDuration="7.099185876s" totalImagesPullingTime="3.423504903s" totalInitContainerRuntime="1.044283558s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:26:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:26:10.097410104 +0000 UTC m=+390.441008053" watchObservedRunningTime="2026-09-09 12:26:10.099185876 +0000 UTC m=+390.442783824"
Sep 09 12:26:21.081181 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:21.081150    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c5d49f968-hfkz2"
Sep 09 12:26:25.455743 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.455711    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"]
Sep 09 12:26:25.466835 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.466811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"
Sep 09 12:26:25.469042 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.469016    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 12:26:25.469252 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.469204    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 12:26:25.469357 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.469272    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w88fg\""
Sep 09 12:26:25.470113 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.470093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"]
Sep 09 12:26:25.596601 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.596568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cfa3ac-1334-4119-9c97-da15303f86ba-tls-certs\") pod \"kube-auth-proxy-5948bbb55b-6bs6z\" (UID: \"c0cfa3ac-1334-4119-9c97-da15303f86ba\") " pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"
Sep 09 12:26:25.596781 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.596608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0cfa3ac-1334-4119-9c97-da15303f86ba-tmp\") pod \"kube-auth-proxy-5948bbb55b-6bs6z\" (UID: \"c0cfa3ac-1334-4119-9c97-da15303f86ba\") " pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"
Sep 09 12:26:25.596781 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.596635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjrz8\" (UniqueName: \"kubernetes.io/projected/c0cfa3ac-1334-4119-9c97-da15303f86ba-kube-api-access-hjrz8\") pod \"kube-auth-proxy-5948bbb55b-6bs6z\" (UID: \"c0cfa3ac-1334-4119-9c97-da15303f86ba\") " pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"
Sep 09 12:26:25.697520 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.697482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrz8\" (UniqueName: \"kubernetes.io/projected/c0cfa3ac-1334-4119-9c97-da15303f86ba-kube-api-access-hjrz8\") pod \"kube-auth-proxy-5948bbb55b-6bs6z\" (UID: \"c0cfa3ac-1334-4119-9c97-da15303f86ba\") " pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"
Sep 09 12:26:25.697697 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.697578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cfa3ac-1334-4119-9c97-da15303f86ba-tls-certs\") pod \"kube-auth-proxy-5948bbb55b-6bs6z\" (UID: \"c0cfa3ac-1334-4119-9c97-da15303f86ba\") " pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"
Sep 09 12:26:25.697697 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.697607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0cfa3ac-1334-4119-9c97-da15303f86ba-tmp\") pod \"kube-auth-proxy-5948bbb55b-6bs6z\" (UID: \"c0cfa3ac-1334-4119-9c97-da15303f86ba\") " pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"
Sep 09 12:26:25.702134 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.700857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0cfa3ac-1334-4119-9c97-da15303f86ba-tmp\") pod \"kube-auth-proxy-5948bbb55b-6bs6z\" (UID: \"c0cfa3ac-1334-4119-9c97-da15303f86ba\") " pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"
Sep 09 12:26:25.702134 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.701016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cfa3ac-1334-4119-9c97-da15303f86ba-tls-certs\") pod \"kube-auth-proxy-5948bbb55b-6bs6z\" (UID: \"c0cfa3ac-1334-4119-9c97-da15303f86ba\") " pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"
Sep 09 12:26:25.708251 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.708163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrz8\" (UniqueName: \"kubernetes.io/projected/c0cfa3ac-1334-4119-9c97-da15303f86ba-kube-api-access-hjrz8\") pod \"kube-auth-proxy-5948bbb55b-6bs6z\" (UID: \"c0cfa3ac-1334-4119-9c97-da15303f86ba\") " pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"
Sep 09 12:26:25.777846 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:25.777809    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"
Sep 09 12:26:26.051612 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:26.051526    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z"]
Sep 09 12:26:26.054492 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:26:26.054468    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cfa3ac_1334_4119_9c97_da15303f86ba.slice/crio-601586473ed4951bef85655104424b1af7e73307d69173aab5e6300126b2ec69 WatchSource:0}: Error finding container 601586473ed4951bef85655104424b1af7e73307d69173aab5e6300126b2ec69: Status 404 returned error can't find the container with id 601586473ed4951bef85655104424b1af7e73307d69173aab5e6300126b2ec69
Sep 09 12:26:26.155872 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:26.155830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z" event={"ID":"c0cfa3ac-1334-4119-9c97-da15303f86ba","Type":"ContainerStarted","Data":"601586473ed4951bef85655104424b1af7e73307d69173aab5e6300126b2ec69"}
Sep 09 12:26:29.176149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:29.176054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z" event={"ID":"c0cfa3ac-1334-4119-9c97-da15303f86ba","Type":"ContainerStarted","Data":"86a505d792c35b7475eabf49825e611730cad610691ffde05ba6a429de5e9ae8"}
Sep 09 12:26:29.193985 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:26:29.193935    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5948bbb55b-6bs6z" podStartSLOduration=1.324813467 podStartE2EDuration="4.193919856s" totalImagesPullingTime="2.869106389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:26:29.19181923 +0000 UTC m=+409.535417178" watchObservedRunningTime="2026-09-09 12:26:29.193919856 +0000 UTC m=+409.537517804"
Sep 09 12:28:03.056845 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.056812    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"]
Sep 09 12:28:03.060360 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.060344    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"
Sep 09 12:28:03.063550 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.063522    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 12:28:03.063550 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.063546    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vb5hj\""
Sep 09 12:28:03.063731 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.063525    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 12:28:03.063731 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.063570    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 12:28:03.063731 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.063587    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 12:28:03.071501 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.071478    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"]
Sep 09 12:28:03.152064 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.152031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ed9b5d1-05db-4345-8dd6-945c00d4b1a0-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-k6t92\" (UID: \"9ed9b5d1-05db-4345-8dd6-945c00d4b1a0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"
Sep 09 12:28:03.152270 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.152089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ed9b5d1-05db-4345-8dd6-945c00d4b1a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-k6t92\" (UID: \"9ed9b5d1-05db-4345-8dd6-945c00d4b1a0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"
Sep 09 12:28:03.152270 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.152179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnrm8\" (UniqueName: \"kubernetes.io/projected/9ed9b5d1-05db-4345-8dd6-945c00d4b1a0-kube-api-access-jnrm8\") pod \"kuadrant-console-plugin-fbdd4f445-k6t92\" (UID: \"9ed9b5d1-05db-4345-8dd6-945c00d4b1a0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"
Sep 09 12:28:03.253519 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.253479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ed9b5d1-05db-4345-8dd6-945c00d4b1a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-k6t92\" (UID: \"9ed9b5d1-05db-4345-8dd6-945c00d4b1a0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"
Sep 09 12:28:03.253715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.253546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnrm8\" (UniqueName: \"kubernetes.io/projected/9ed9b5d1-05db-4345-8dd6-945c00d4b1a0-kube-api-access-jnrm8\") pod \"kuadrant-console-plugin-fbdd4f445-k6t92\" (UID: \"9ed9b5d1-05db-4345-8dd6-945c00d4b1a0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"
Sep 09 12:28:03.253715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.253598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ed9b5d1-05db-4345-8dd6-945c00d4b1a0-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-k6t92\" (UID: \"9ed9b5d1-05db-4345-8dd6-945c00d4b1a0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"
Sep 09 12:28:03.254326 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.254307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ed9b5d1-05db-4345-8dd6-945c00d4b1a0-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-k6t92\" (UID: \"9ed9b5d1-05db-4345-8dd6-945c00d4b1a0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"
Sep 09 12:28:03.256144 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.256120    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ed9b5d1-05db-4345-8dd6-945c00d4b1a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-k6t92\" (UID: \"9ed9b5d1-05db-4345-8dd6-945c00d4b1a0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"
Sep 09 12:28:03.261227 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.261195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnrm8\" (UniqueName: \"kubernetes.io/projected/9ed9b5d1-05db-4345-8dd6-945c00d4b1a0-kube-api-access-jnrm8\") pod \"kuadrant-console-plugin-fbdd4f445-k6t92\" (UID: \"9ed9b5d1-05db-4345-8dd6-945c00d4b1a0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"
Sep 09 12:28:03.383644 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.383567    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"
Sep 09 12:28:03.656704 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:03.656679    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92"]
Sep 09 12:28:03.659352 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:28:03.659323    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ed9b5d1_05db_4345_8dd6_945c00d4b1a0.slice/crio-ad5214c7881b2ebb4c8712b26e323dc30658fe345194a0bca068f881b589a2e3 WatchSource:0}: Error finding container ad5214c7881b2ebb4c8712b26e323dc30658fe345194a0bca068f881b589a2e3: Status 404 returned error can't find the container with id ad5214c7881b2ebb4c8712b26e323dc30658fe345194a0bca068f881b589a2e3
Sep 09 12:28:04.616817 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:04.616785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92" event={"ID":"9ed9b5d1-05db-4345-8dd6-945c00d4b1a0","Type":"ContainerStarted","Data":"ad5214c7881b2ebb4c8712b26e323dc30658fe345194a0bca068f881b589a2e3"}
Sep 09 12:28:08.638487 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:08.638454    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92" event={"ID":"9ed9b5d1-05db-4345-8dd6-945c00d4b1a0","Type":"ContainerStarted","Data":"334d8a0a6b7d98311ec01f4c7ffb18f1774a06c42469c32fb3fd80aa3f3b7b82"}
Sep 09 12:28:08.654154 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:08.654107    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k6t92" podStartSLOduration=0.815877895 podStartE2EDuration="5.654092003s" totalImagesPullingTime="4.838214108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:28:08.653771489 +0000 UTC m=+508.997369450" watchObservedRunningTime="2026-09-09 12:28:08.654092003 +0000 UTC m=+508.997689951"
Sep 09 12:28:56.676972 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:56.676895    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:28:56.695314 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:56.695282    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:28:56.695467 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:56.695387    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"
Sep 09 12:28:56.697383 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:56.697364    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 12:28:56.704082 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:56.704056    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:28:56.827248 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:56.827182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4451aeda-66e4-4ad3-a46a-5084e4e7a738-config-file\") pod \"limitador-limitador-84994f7ffc-jrc6g\" (UID: \"4451aeda-66e4-4ad3-a46a-5084e4e7a738\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"
Sep 09 12:28:56.827430 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:56.827308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfcb\" (UniqueName: \"kubernetes.io/projected/4451aeda-66e4-4ad3-a46a-5084e4e7a738-kube-api-access-zjfcb\") pod \"limitador-limitador-84994f7ffc-jrc6g\" (UID: \"4451aeda-66e4-4ad3-a46a-5084e4e7a738\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"
Sep 09 12:28:56.928847 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:56.928752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4451aeda-66e4-4ad3-a46a-5084e4e7a738-config-file\") pod \"limitador-limitador-84994f7ffc-jrc6g\" (UID: \"4451aeda-66e4-4ad3-a46a-5084e4e7a738\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"
Sep 09 12:28:56.928847 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:56.928810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfcb\" (UniqueName: \"kubernetes.io/projected/4451aeda-66e4-4ad3-a46a-5084e4e7a738-kube-api-access-zjfcb\") pod \"limitador-limitador-84994f7ffc-jrc6g\" (UID: \"4451aeda-66e4-4ad3-a46a-5084e4e7a738\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"
Sep 09 12:28:56.929461 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:56.929438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4451aeda-66e4-4ad3-a46a-5084e4e7a738-config-file\") pod \"limitador-limitador-84994f7ffc-jrc6g\" (UID: \"4451aeda-66e4-4ad3-a46a-5084e4e7a738\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"
Sep 09 12:28:56.936731 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:56.936707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfcb\" (UniqueName: \"kubernetes.io/projected/4451aeda-66e4-4ad3-a46a-5084e4e7a738-kube-api-access-zjfcb\") pod \"limitador-limitador-84994f7ffc-jrc6g\" (UID: \"4451aeda-66e4-4ad3-a46a-5084e4e7a738\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"
Sep 09 12:28:57.006463 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.006419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"
Sep 09 12:28:57.237352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.237273    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-wzwz8"]
Sep 09 12:28:57.240647 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.240632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wzwz8"
Sep 09 12:28:57.242500 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.242481    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lsjjv\""
Sep 09 12:28:57.245929 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.245897    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wzwz8"]
Sep 09 12:28:57.277077 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.277050    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:28:57.279752 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:28:57.279724    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4451aeda_66e4_4ad3_a46a_5084e4e7a738.slice/crio-79549e01a9da5c50d9fae0488d6a50ce69dfa3bef4bfed245a862c1709e0c961 WatchSource:0}: Error finding container 79549e01a9da5c50d9fae0488d6a50ce69dfa3bef4bfed245a862c1709e0c961: Status 404 returned error can't find the container with id 79549e01a9da5c50d9fae0488d6a50ce69dfa3bef4bfed245a862c1709e0c961
Sep 09 12:28:57.332128 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.332092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ztf\" (UniqueName: \"kubernetes.io/projected/48ef0d97-f2a5-4a24-b7bf-e31482938f73-kube-api-access-66ztf\") pod \"authorino-59c974bd48-wzwz8\" (UID: \"48ef0d97-f2a5-4a24-b7bf-e31482938f73\") " pod="kuadrant-system/authorino-59c974bd48-wzwz8"
Sep 09 12:28:57.433612 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.433581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ztf\" (UniqueName: \"kubernetes.io/projected/48ef0d97-f2a5-4a24-b7bf-e31482938f73-kube-api-access-66ztf\") pod \"authorino-59c974bd48-wzwz8\" (UID: \"48ef0d97-f2a5-4a24-b7bf-e31482938f73\") " pod="kuadrant-system/authorino-59c974bd48-wzwz8"
Sep 09 12:28:57.441990 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.441957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ztf\" (UniqueName: \"kubernetes.io/projected/48ef0d97-f2a5-4a24-b7bf-e31482938f73-kube-api-access-66ztf\") pod \"authorino-59c974bd48-wzwz8\" (UID: \"48ef0d97-f2a5-4a24-b7bf-e31482938f73\") " pod="kuadrant-system/authorino-59c974bd48-wzwz8"
Sep 09 12:28:57.551433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.551348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wzwz8"
Sep 09 12:28:57.817145 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.817117    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wzwz8"]
Sep 09 12:28:57.819180 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:28:57.819157    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48ef0d97_f2a5_4a24_b7bf_e31482938f73.slice/crio-e793a64c8f5ce002c756477bcf7c5b447f14e9755b636c7de18fb88f06aa5c2e WatchSource:0}: Error finding container e793a64c8f5ce002c756477bcf7c5b447f14e9755b636c7de18fb88f06aa5c2e: Status 404 returned error can't find the container with id e793a64c8f5ce002c756477bcf7c5b447f14e9755b636c7de18fb88f06aa5c2e
Sep 09 12:28:57.877444 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.877403    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g" event={"ID":"4451aeda-66e4-4ad3-a46a-5084e4e7a738","Type":"ContainerStarted","Data":"79549e01a9da5c50d9fae0488d6a50ce69dfa3bef4bfed245a862c1709e0c961"}
Sep 09 12:28:57.878657 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:28:57.878634    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wzwz8" event={"ID":"48ef0d97-f2a5-4a24-b7bf-e31482938f73","Type":"ContainerStarted","Data":"e793a64c8f5ce002c756477bcf7c5b447f14e9755b636c7de18fb88f06aa5c2e"}
Sep 09 12:29:00.897054 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:00.897013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g" event={"ID":"4451aeda-66e4-4ad3-a46a-5084e4e7a738","Type":"ContainerStarted","Data":"6e8b2561c844b3eae28f0f04ae9ff6e62501ea0a51bf56c5b36c07fc16140fbf"}
Sep 09 12:29:00.897514 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:00.897125    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"
Sep 09 12:29:00.898485 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:00.898463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wzwz8" event={"ID":"48ef0d97-f2a5-4a24-b7bf-e31482938f73","Type":"ContainerStarted","Data":"133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a"}
Sep 09 12:29:00.914591 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:00.914535    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g" podStartSLOduration=1.949989332 podStartE2EDuration="4.914517034s" totalImagesPullingTime="2.964527702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:29:00.914027049 +0000 UTC m=+561.257624998" watchObservedRunningTime="2026-09-09 12:29:00.914517034 +0000 UTC m=+561.258114983"
Sep 09 12:29:00.932760 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:00.932701    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-wzwz8" podStartSLOduration=1.501493121 podStartE2EDuration="3.93268741s" totalImagesPullingTime="2.431194289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:29:00.932653024 +0000 UTC m=+561.276250965" watchObservedRunningTime="2026-09-09 12:29:00.93268741 +0000 UTC m=+561.276285357"
Sep 09 12:29:11.904344 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:11.904313    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"
Sep 09 12:29:27.475669 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:27.475632    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65f7969957-pfk7h"]
Sep 09 12:29:27.479233 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:27.479189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f7969957-pfk7h"
Sep 09 12:29:27.481194 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:27.481176    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 12:29:27.486345 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:27.486324    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65f7969957-pfk7h"]
Sep 09 12:29:27.510895 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:27.510867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkfkc\" (UniqueName: \"kubernetes.io/projected/511d5b66-d853-44ef-ac2f-595bb3cb534f-kube-api-access-jkfkc\") pod \"authorino-65f7969957-pfk7h\" (UID: \"511d5b66-d853-44ef-ac2f-595bb3cb534f\") " pod="kuadrant-system/authorino-65f7969957-pfk7h"
Sep 09 12:29:27.511028 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:27.510908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/511d5b66-d853-44ef-ac2f-595bb3cb534f-tls-cert\") pod \"authorino-65f7969957-pfk7h\" (UID: \"511d5b66-d853-44ef-ac2f-595bb3cb534f\") " pod="kuadrant-system/authorino-65f7969957-pfk7h"
Sep 09 12:29:27.612328 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:27.612286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfkc\" (UniqueName: \"kubernetes.io/projected/511d5b66-d853-44ef-ac2f-595bb3cb534f-kube-api-access-jkfkc\") pod \"authorino-65f7969957-pfk7h\" (UID: \"511d5b66-d853-44ef-ac2f-595bb3cb534f\") " pod="kuadrant-system/authorino-65f7969957-pfk7h"
Sep 09 12:29:27.612580 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:27.612348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/511d5b66-d853-44ef-ac2f-595bb3cb534f-tls-cert\") pod \"authorino-65f7969957-pfk7h\" (UID: \"511d5b66-d853-44ef-ac2f-595bb3cb534f\") " pod="kuadrant-system/authorino-65f7969957-pfk7h"
Sep 09 12:29:27.614961 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:27.614941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/511d5b66-d853-44ef-ac2f-595bb3cb534f-tls-cert\") pod \"authorino-65f7969957-pfk7h\" (UID: \"511d5b66-d853-44ef-ac2f-595bb3cb534f\") " pod="kuadrant-system/authorino-65f7969957-pfk7h"
Sep 09 12:29:27.625554 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:27.625529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkfkc\" (UniqueName: \"kubernetes.io/projected/511d5b66-d853-44ef-ac2f-595bb3cb534f-kube-api-access-jkfkc\") pod \"authorino-65f7969957-pfk7h\" (UID: \"511d5b66-d853-44ef-ac2f-595bb3cb534f\") " pod="kuadrant-system/authorino-65f7969957-pfk7h"
Sep 09 12:29:27.788553 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:27.788463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f7969957-pfk7h"
Sep 09 12:29:28.057451 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:28.057425    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65f7969957-pfk7h"]
Sep 09 12:29:28.059580 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:29:28.059547    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod511d5b66_d853_44ef_ac2f_595bb3cb534f.slice/crio-a3c4e15b0df6e8f061d85904b9b2f2ec788d6eb7ee5c9bb5828f7d7fb947d529 WatchSource:0}: Error finding container a3c4e15b0df6e8f061d85904b9b2f2ec788d6eb7ee5c9bb5828f7d7fb947d529: Status 404 returned error can't find the container with id a3c4e15b0df6e8f061d85904b9b2f2ec788d6eb7ee5c9bb5828f7d7fb947d529
Sep 09 12:29:29.035809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:29.035714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f7969957-pfk7h" event={"ID":"511d5b66-d853-44ef-ac2f-595bb3cb534f","Type":"ContainerStarted","Data":"aa9c34c79c7c3d954fc7ad108635cd1c0e70956081783aa836eab702e2517d89"}
Sep 09 12:29:29.035809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:29.035759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f7969957-pfk7h" event={"ID":"511d5b66-d853-44ef-ac2f-595bb3cb534f","Type":"ContainerStarted","Data":"a3c4e15b0df6e8f061d85904b9b2f2ec788d6eb7ee5c9bb5828f7d7fb947d529"}
Sep 09 12:29:29.052293 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:29.052241    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65f7969957-pfk7h" podStartSLOduration=1.445380599 podStartE2EDuration="2.052209254s" totalImagesPullingTime="606.828655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:29:29.051028271 +0000 UTC m=+589.394626219" watchObservedRunningTime="2026-09-09 12:29:29.052209254 +0000 UTC m=+589.395807202"
Sep 09 12:29:29.080141 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:29.080105    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wzwz8"]
Sep 09 12:29:29.080332 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:29.080312    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-wzwz8" podUID="48ef0d97-f2a5-4a24-b7bf-e31482938f73" containerName="authorino" containerID="cri-o://133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a" gracePeriod=30
Sep 09 12:29:29.330895 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:29.330871    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wzwz8"
Sep 09 12:29:29.426565 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:29.426526    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66ztf\" (UniqueName: \"kubernetes.io/projected/48ef0d97-f2a5-4a24-b7bf-e31482938f73-kube-api-access-66ztf\") pod \"48ef0d97-f2a5-4a24-b7bf-e31482938f73\" (UID: \"48ef0d97-f2a5-4a24-b7bf-e31482938f73\") "
Sep 09 12:29:29.428940 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:29.428915    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ef0d97-f2a5-4a24-b7bf-e31482938f73-kube-api-access-66ztf" (OuterVolumeSpecName: "kube-api-access-66ztf") pod "48ef0d97-f2a5-4a24-b7bf-e31482938f73" (UID: "48ef0d97-f2a5-4a24-b7bf-e31482938f73"). InnerVolumeSpecName "kube-api-access-66ztf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:29:29.527860 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:29.527829    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66ztf\" (UniqueName: \"kubernetes.io/projected/48ef0d97-f2a5-4a24-b7bf-e31482938f73-kube-api-access-66ztf\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:29:30.041914 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:30.041819    2578 generic.go:356] "Generic (PLEG): container finished" podID="48ef0d97-f2a5-4a24-b7bf-e31482938f73" containerID="133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a" exitCode=0
Sep 09 12:29:30.041914 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:30.041859    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wzwz8"
Sep 09 12:29:30.042425 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:30.041910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wzwz8" event={"ID":"48ef0d97-f2a5-4a24-b7bf-e31482938f73","Type":"ContainerDied","Data":"133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a"}
Sep 09 12:29:30.042425 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:30.041947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wzwz8" event={"ID":"48ef0d97-f2a5-4a24-b7bf-e31482938f73","Type":"ContainerDied","Data":"e793a64c8f5ce002c756477bcf7c5b447f14e9755b636c7de18fb88f06aa5c2e"}
Sep 09 12:29:30.042425 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:30.041968    2578 scope.go:117] "RemoveContainer" containerID="133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a"
Sep 09 12:29:30.051199 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:30.051176    2578 scope.go:117] "RemoveContainer" containerID="133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a"
Sep 09 12:29:30.051564 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:29:30.051538    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a\": container with ID starting with 133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a not found: ID does not exist" containerID="133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a"
Sep 09 12:29:30.051650 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:30.051571    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a"} err="failed to get container status \"133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a\": rpc error: code = NotFound desc = could not find container \"133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a\": container with ID starting with 133bdc940dd860d157f4d3f7cac03bbaceafc0673ac554b655c60718120e2a9a not found: ID does not exist"
Sep 09 12:29:30.062772 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:30.062748    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wzwz8"]
Sep 09 12:29:30.065946 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:30.065920    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wzwz8"]
Sep 09 12:29:30.185652 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:30.185623    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48ef0d97-f2a5-4a24-b7bf-e31482938f73" path="/var/lib/kubelet/pods/48ef0d97-f2a5-4a24-b7bf-e31482938f73/volumes"
Sep 09 12:29:40.119143 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:40.119112    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:29:40.121182 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:29:40.121153    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:30:12.312837 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:12.312752    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:30:32.835294 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:32.835264    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:30:41.429894 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:41.429860    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:30:46.027296 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:46.027256    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:30:55.023173 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.023135    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"]
Sep 09 12:30:55.023653 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.023636    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ef0d97-f2a5-4a24-b7bf-e31482938f73" containerName="authorino"
Sep 09 12:30:55.023653 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.023651    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ef0d97-f2a5-4a24-b7bf-e31482938f73" containerName="authorino"
Sep 09 12:30:55.023740 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.023728    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ef0d97-f2a5-4a24-b7bf-e31482938f73" containerName="authorino"
Sep 09 12:30:55.026916 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.026902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.029656 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.029636    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 12:30:55.029786 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.029637    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 12:30:55.029786 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.029638    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2r6ws\""
Sep 09 12:30:55.029786 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.029682    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 12:30:55.049133 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.049107    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"]
Sep 09 12:30:55.090937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.090906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.090937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.090942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.091152 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.090967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.091152 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.091047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.091152 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.091101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6nsr\" (UniqueName: \"kubernetes.io/projected/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-kube-api-access-v6nsr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.091152 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.091147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.192448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.192411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6nsr\" (UniqueName: \"kubernetes.io/projected/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-kube-api-access-v6nsr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.192448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.192450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.192683 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.192515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.192683 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.192535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.192683 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.192552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.192683 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.192581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.193023 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.192962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.193111 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.193030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.193111 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.193062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.195128 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.195111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.195194 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.195174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.200240 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.200199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6nsr\" (UniqueName: \"kubernetes.io/projected/99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4-kube-api-access-v6nsr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxhq\" (UID: \"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.336998 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.336912    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:30:55.611933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.611906    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"]
Sep 09 12:30:55.614458 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:30:55.614429    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e5f97e_ed4b_44a2_b4f1_0ac45e61f8e4.slice/crio-89c98d2030ee12dd661d6d4bf111e2769393aba27e0d666cfbdf0c1c9cc42a73 WatchSource:0}: Error finding container 89c98d2030ee12dd661d6d4bf111e2769393aba27e0d666cfbdf0c1c9cc42a73: Status 404 returned error can't find the container with id 89c98d2030ee12dd661d6d4bf111e2769393aba27e0d666cfbdf0c1c9cc42a73
Sep 09 12:30:55.616109 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:55.616088    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:30:56.336144 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:56.336103    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:30:56.463126 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:56.463090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq" event={"ID":"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4","Type":"ContainerStarted","Data":"89c98d2030ee12dd661d6d4bf111e2769393aba27e0d666cfbdf0c1c9cc42a73"}
Sep 09 12:30:57.468864 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:57.468834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq" event={"ID":"99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4","Type":"ContainerStarted","Data":"d95b5d199107d9d9d724797472df6b65a1b29552866b4894edfa289011b38b18"}
Sep 09 12:30:57.492737 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:30:57.492686    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq" podStartSLOduration=1.448793646 podStartE2EDuration="2.492670251s" totalImagesPullingTime="1.043876605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:30:57.491074608 +0000 UTC m=+677.834672558" watchObservedRunningTime="2026-09-09 12:30:57.492670251 +0000 UTC m=+677.836268199"
Sep 09 12:31:05.337319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:05.337280    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:31:05.337711 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:05.337345    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:31:05.349996 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:05.349970    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:31:05.519946 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:05.519922    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxhq"
Sep 09 12:31:21.435308 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:21.435270    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:31:29.334207 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.334175    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"]
Sep 09 12:31:29.339438 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.339416    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.341416 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.341395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 12:31:29.347561 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.347541    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"]
Sep 09 12:31:29.424872 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.424832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.424872 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.424874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.425077 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.424915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c383e6-88a2-44ab-a757-e484818d9b04-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.425077 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.424967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.425077 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.425040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.425077 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.425065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q49sh\" (UniqueName: \"kubernetes.io/projected/13c383e6-88a2-44ab-a757-e484818d9b04-kube-api-access-q49sh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.525609 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.525560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.525609 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.525614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q49sh\" (UniqueName: \"kubernetes.io/projected/13c383e6-88a2-44ab-a757-e484818d9b04-kube-api-access-q49sh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.525843 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.525666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.525843 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.525683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.525946 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.525889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c383e6-88a2-44ab-a757-e484818d9b04-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.526001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.525941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.526101 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.526076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.526164 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.526136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.526233 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.526201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.528026 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.528000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c383e6-88a2-44ab-a757-e484818d9b04-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.528387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.528367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c383e6-88a2-44ab-a757-e484818d9b04-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.534018 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.533994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q49sh\" (UniqueName: \"kubernetes.io/projected/13c383e6-88a2-44ab-a757-e484818d9b04-kube-api-access-q49sh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-88k2b\" (UID: \"13c383e6-88a2-44ab-a757-e484818d9b04\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.650417 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.650385    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:29.925195 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:29.925173    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"]
Sep 09 12:31:29.927176 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:31:29.927146    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c383e6_88a2_44ab_a757_e484818d9b04.slice/crio-5db9909158e5284c2728d16932891af6ece3624f4e86bbf44831312cb73e7cf8 WatchSource:0}: Error finding container 5db9909158e5284c2728d16932891af6ece3624f4e86bbf44831312cb73e7cf8: Status 404 returned error can't find the container with id 5db9909158e5284c2728d16932891af6ece3624f4e86bbf44831312cb73e7cf8
Sep 09 12:31:30.637001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:30.636961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b" event={"ID":"13c383e6-88a2-44ab-a757-e484818d9b04","Type":"ContainerStarted","Data":"1093cd6d8734e48c50289fb4fe43b5eafdb206d701b466bd0c5d706f56b90e6c"}
Sep 09 12:31:30.637413 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:30.637049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b" event={"ID":"13c383e6-88a2-44ab-a757-e484818d9b04","Type":"ContainerStarted","Data":"5db9909158e5284c2728d16932891af6ece3624f4e86bbf44831312cb73e7cf8"}
Sep 09 12:31:30.657451 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:30.657393    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b" podStartSLOduration=1.472118658 podStartE2EDuration="1.657378111s" totalImagesPullingTime="185.259453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:31:30.656141432 +0000 UTC m=+710.999739380" watchObservedRunningTime="2026-09-09 12:31:30.657378111 +0000 UTC m=+711.000976059"
Sep 09 12:31:31.336325 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:31.336230    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:31:39.650596 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:39.650553    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:39.651011 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:39.650644    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:39.663417 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:39.663384    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:39.690364 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:39.690330    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-88k2b"
Sep 09 12:31:43.731748 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.731710    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"]
Sep 09 12:31:43.736075 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.736051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.738526 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.738502    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 12:31:43.748716 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.748689    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"]
Sep 09 12:31:43.861837 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.861802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d17903d-f84a-40ec-8289-a9a431ddec10-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.862012 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.861853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5k44\" (UniqueName: \"kubernetes.io/projected/8d17903d-f84a-40ec-8289-a9a431ddec10-kube-api-access-h5k44\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.862012 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.861917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.862012 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.861942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.862012 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.861992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.862161 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.862048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.963527 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.963493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.963693 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.963540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d17903d-f84a-40ec-8289-a9a431ddec10-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.963693 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.963583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5k44\" (UniqueName: \"kubernetes.io/projected/8d17903d-f84a-40ec-8289-a9a431ddec10-kube-api-access-h5k44\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.963693 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.963612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.963693 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.963629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.963693 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.963688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.963997 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.963966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.964045 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.964021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.964168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.964144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.966130 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.966113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d17903d-f84a-40ec-8289-a9a431ddec10-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.966306 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.966288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d17903d-f84a-40ec-8289-a9a431ddec10-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:43.971813 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:43.971793    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5k44\" (UniqueName: \"kubernetes.io/projected/8d17903d-f84a-40ec-8289-a9a431ddec10-kube-api-access-h5k44\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4\" (UID: \"8d17903d-f84a-40ec-8289-a9a431ddec10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:44.048959 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:44.048877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:44.328525 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:44.328499    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"]
Sep 09 12:31:44.331029 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:31:44.331002    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d17903d_f84a_40ec_8289_a9a431ddec10.slice/crio-3f9c13e6877c3535b49e2959099d3edadf6e0dfb207294fda729a7af0e567a34 WatchSource:0}: Error finding container 3f9c13e6877c3535b49e2959099d3edadf6e0dfb207294fda729a7af0e567a34: Status 404 returned error can't find the container with id 3f9c13e6877c3535b49e2959099d3edadf6e0dfb207294fda729a7af0e567a34
Sep 09 12:31:44.704735 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:44.704694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4" event={"ID":"8d17903d-f84a-40ec-8289-a9a431ddec10","Type":"ContainerStarted","Data":"5951ce55c655cc6b3ac9cd9d9c6cae752e56d1687aa9e64d88c2219c0cbf59db"}
Sep 09 12:31:44.704735 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:44.704734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4" event={"ID":"8d17903d-f84a-40ec-8289-a9a431ddec10","Type":"ContainerStarted","Data":"3f9c13e6877c3535b49e2959099d3edadf6e0dfb207294fda729a7af0e567a34"}
Sep 09 12:31:44.722992 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:44.722935    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4" podStartSLOduration=1.530045731 podStartE2EDuration="1.722919317s" totalImagesPullingTime="192.873586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:31:44.722041218 +0000 UTC m=+725.065639167" watchObservedRunningTime="2026-09-09 12:31:44.722919317 +0000 UTC m=+725.066517266"
Sep 09 12:31:46.132859 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:46.132826    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:31:54.049362 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:54.049311    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:54.049362 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:54.049373    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:54.061862 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:54.061834    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:31:54.763075 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:31:54.763048    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4"
Sep 09 12:32:52.887446 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:52.887399    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65f7969957-pfk7h"]
Sep 09 12:32:52.888083 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:52.887595    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-65f7969957-pfk7h" podUID="511d5b66-d853-44ef-ac2f-595bb3cb534f" containerName="authorino" containerID="cri-o://aa9c34c79c7c3d954fc7ad108635cd1c0e70956081783aa836eab702e2517d89" gracePeriod=30
Sep 09 12:32:53.032970 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:53.032867    2578 generic.go:356] "Generic (PLEG): container finished" podID="511d5b66-d853-44ef-ac2f-595bb3cb534f" containerID="aa9c34c79c7c3d954fc7ad108635cd1c0e70956081783aa836eab702e2517d89" exitCode=0
Sep 09 12:32:53.032970 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:53.032912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f7969957-pfk7h" event={"ID":"511d5b66-d853-44ef-ac2f-595bb3cb534f","Type":"ContainerDied","Data":"aa9c34c79c7c3d954fc7ad108635cd1c0e70956081783aa836eab702e2517d89"}
Sep 09 12:32:53.141968 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:53.141912    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f7969957-pfk7h"
Sep 09 12:32:53.229150 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:53.229096    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/511d5b66-d853-44ef-ac2f-595bb3cb534f-tls-cert\") pod \"511d5b66-d853-44ef-ac2f-595bb3cb534f\" (UID: \"511d5b66-d853-44ef-ac2f-595bb3cb534f\") "
Sep 09 12:32:53.229358 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:53.229287    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkfkc\" (UniqueName: \"kubernetes.io/projected/511d5b66-d853-44ef-ac2f-595bb3cb534f-kube-api-access-jkfkc\") pod \"511d5b66-d853-44ef-ac2f-595bb3cb534f\" (UID: \"511d5b66-d853-44ef-ac2f-595bb3cb534f\") "
Sep 09 12:32:53.231361 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:53.231326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511d5b66-d853-44ef-ac2f-595bb3cb534f-kube-api-access-jkfkc" (OuterVolumeSpecName: "kube-api-access-jkfkc") pod "511d5b66-d853-44ef-ac2f-595bb3cb534f" (UID: "511d5b66-d853-44ef-ac2f-595bb3cb534f"). InnerVolumeSpecName "kube-api-access-jkfkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:32:53.239613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:53.239586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511d5b66-d853-44ef-ac2f-595bb3cb534f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "511d5b66-d853-44ef-ac2f-595bb3cb534f" (UID: "511d5b66-d853-44ef-ac2f-595bb3cb534f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:32:53.330419 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:53.330362    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkfkc\" (UniqueName: \"kubernetes.io/projected/511d5b66-d853-44ef-ac2f-595bb3cb534f-kube-api-access-jkfkc\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:32:53.330419 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:53.330411    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/511d5b66-d853-44ef-ac2f-595bb3cb534f-tls-cert\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:32:54.039605 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:54.039567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f7969957-pfk7h" event={"ID":"511d5b66-d853-44ef-ac2f-595bb3cb534f","Type":"ContainerDied","Data":"a3c4e15b0df6e8f061d85904b9b2f2ec788d6eb7ee5c9bb5828f7d7fb947d529"}
Sep 09 12:32:54.039605 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:54.039605    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f7969957-pfk7h"
Sep 09 12:32:54.040119 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:54.039625    2578 scope.go:117] "RemoveContainer" containerID="aa9c34c79c7c3d954fc7ad108635cd1c0e70956081783aa836eab702e2517d89"
Sep 09 12:32:54.062896 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:54.062865    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65f7969957-pfk7h"]
Sep 09 12:32:54.069806 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:54.069777    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65f7969957-pfk7h"]
Sep 09 12:32:54.186520 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:32:54.186492    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="511d5b66-d853-44ef-ac2f-595bb3cb534f" path="/var/lib/kubelet/pods/511d5b66-d853-44ef-ac2f-595bb3cb534f/volumes"
Sep 09 12:33:12.566760 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:12.566666    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-d0b62a"]
Sep 09 12:33:12.567183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:12.567088    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511d5b66-d853-44ef-ac2f-595bb3cb534f" containerName="authorino"
Sep 09 12:33:12.567183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:12.567100    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="511d5b66-d853-44ef-ac2f-595bb3cb534f" containerName="authorino"
Sep 09 12:33:12.567183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:12.567172    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="511d5b66-d853-44ef-ac2f-595bb3cb534f" containerName="authorino"
Sep 09 12:33:12.570130 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:12.570114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-d0b62a"
Sep 09 12:33:12.572251 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:12.572198    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4dspl\""
Sep 09 12:33:12.578444 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:12.577748    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-d0b62a"]
Sep 09 12:33:12.713382 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:12.713347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4dd6\" (UniqueName: \"kubernetes.io/projected/7398f5be-078b-41e5-ab2e-bbe53c096e41-kube-api-access-t4dd6\") pod \"test-curl-7684-d0b62a\" (UID: \"7398f5be-078b-41e5-ab2e-bbe53c096e41\") " pod="opendatahub/test-curl-7684-d0b62a"
Sep 09 12:33:12.814842 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:12.814798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4dd6\" (UniqueName: \"kubernetes.io/projected/7398f5be-078b-41e5-ab2e-bbe53c096e41-kube-api-access-t4dd6\") pod \"test-curl-7684-d0b62a\" (UID: \"7398f5be-078b-41e5-ab2e-bbe53c096e41\") " pod="opendatahub/test-curl-7684-d0b62a"
Sep 09 12:33:12.823745 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:12.823687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4dd6\" (UniqueName: \"kubernetes.io/projected/7398f5be-078b-41e5-ab2e-bbe53c096e41-kube-api-access-t4dd6\") pod \"test-curl-7684-d0b62a\" (UID: \"7398f5be-078b-41e5-ab2e-bbe53c096e41\") " pod="opendatahub/test-curl-7684-d0b62a"
Sep 09 12:33:12.881447 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:12.881397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-d0b62a"
Sep 09 12:33:13.150572 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:13.150545    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-d0b62a"]
Sep 09 12:33:13.154423 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:33:13.154397    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7398f5be_078b_41e5_ab2e_bbe53c096e41.slice/crio-551b903a74fde4b3c7008eb77d716cd618bd6be1b91b402accf4943ce8c79cf3 WatchSource:0}: Error finding container 551b903a74fde4b3c7008eb77d716cd618bd6be1b91b402accf4943ce8c79cf3: Status 404 returned error can't find the container with id 551b903a74fde4b3c7008eb77d716cd618bd6be1b91b402accf4943ce8c79cf3
Sep 09 12:33:14.147579 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:14.147524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-d0b62a" event={"ID":"7398f5be-078b-41e5-ab2e-bbe53c096e41","Type":"ContainerStarted","Data":"551b903a74fde4b3c7008eb77d716cd618bd6be1b91b402accf4943ce8c79cf3"}
Sep 09 12:33:14.537448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:14.537413    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:33:16.160055 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:16.160022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-d0b62a" event={"ID":"7398f5be-078b-41e5-ab2e-bbe53c096e41","Type":"ContainerStarted","Data":"99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900"}
Sep 09 12:33:16.177643 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:16.177590    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7684-d0b62a" podStartSLOduration=1.9121145739999998 podStartE2EDuration="4.177574974s" totalImagesPullingTime="2.2654604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:33:16.176990765 +0000 UTC m=+816.520588716" watchObservedRunningTime="2026-09-09 12:33:16.177574974 +0000 UTC m=+816.521172921"
Sep 09 12:33:16.758942 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:16.758906    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-d0b62a"]
Sep 09 12:33:16.762929 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:16.762904    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-d0b62a"]
Sep 09 12:33:16.883698 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:16.883622    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-4edb6a"]
Sep 09 12:33:16.883998 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:16.883986    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7398f5be-078b-41e5-ab2e-bbe53c096e41" containerName="test-curl-7684-d0b62a"
Sep 09 12:33:16.884041 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:16.884000    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7398f5be-078b-41e5-ab2e-bbe53c096e41" containerName="test-curl-7684-d0b62a"
Sep 09 12:33:16.884075 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:16.884069    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7398f5be-078b-41e5-ab2e-bbe53c096e41" containerName="test-curl-7684-d0b62a"
Sep 09 12:33:16.888873 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:16.888846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-4edb6a"
Sep 09 12:33:16.894723 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:16.894697    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-4edb6a"]
Sep 09 12:33:17.061274 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:17.061235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkctt\" (UniqueName: \"kubernetes.io/projected/642e1d38-1ad2-4c81-beb6-f2c98bcd9b29-kube-api-access-fkctt\") pod \"test-curl-7684-4edb6a\" (UID: \"642e1d38-1ad2-4c81-beb6-f2c98bcd9b29\") " pod="opendatahub/test-curl-7684-4edb6a"
Sep 09 12:33:17.162588 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:17.162557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkctt\" (UniqueName: \"kubernetes.io/projected/642e1d38-1ad2-4c81-beb6-f2c98bcd9b29-kube-api-access-fkctt\") pod \"test-curl-7684-4edb6a\" (UID: \"642e1d38-1ad2-4c81-beb6-f2c98bcd9b29\") " pod="opendatahub/test-curl-7684-4edb6a"
Sep 09 12:33:17.165293 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:17.165273    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7684-d0b62a" container="test-curl-7684-d0b62a"
Sep 09 12:33:17.171410 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:17.171379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkctt\" (UniqueName: \"kubernetes.io/projected/642e1d38-1ad2-4c81-beb6-f2c98bcd9b29-kube-api-access-fkctt\") pod \"test-curl-7684-4edb6a\" (UID: \"642e1d38-1ad2-4c81-beb6-f2c98bcd9b29\") " pod="opendatahub/test-curl-7684-4edb6a"
Sep 09 12:33:17.200024 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:17.200001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-4edb6a"
Sep 09 12:33:17.473557 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:17.473531    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-4edb6a"]
Sep 09 12:33:17.476269 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:33:17.476239    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod642e1d38_1ad2_4c81_beb6_f2c98bcd9b29.slice/crio-0b6b59eda48e543ea07891f6bb5e972d0880c218804f911a8df711dcf334fd16 WatchSource:0}: Error finding container 0b6b59eda48e543ea07891f6bb5e972d0880c218804f911a8df711dcf334fd16: Status 404 returned error can't find the container with id 0b6b59eda48e543ea07891f6bb5e972d0880c218804f911a8df711dcf334fd16
Sep 09 12:33:18.171853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:18.171814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-4edb6a" event={"ID":"642e1d38-1ad2-4c81-beb6-f2c98bcd9b29","Type":"ContainerStarted","Data":"81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f"}
Sep 09 12:33:18.171853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:18.171860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-4edb6a" event={"ID":"642e1d38-1ad2-4c81-beb6-f2c98bcd9b29","Type":"ContainerStarted","Data":"0b6b59eda48e543ea07891f6bb5e972d0880c218804f911a8df711dcf334fd16"}
Sep 09 12:33:18.172380 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:18.171966    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7684-d0b62a" podUID="7398f5be-078b-41e5-ab2e-bbe53c096e41" containerName="test-curl-7684-d0b62a" containerID="cri-o://99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900" gracePeriod=2
Sep 09 12:33:18.187555 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:18.187513    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7684-4edb6a" podStartSLOduration=2.187497288 podStartE2EDuration="2.187497288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:33:18.186822864 +0000 UTC m=+818.530420810" watchObservedRunningTime="2026-09-09 12:33:18.187497288 +0000 UTC m=+818.531095235"
Sep 09 12:33:18.769482 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:18.769448    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-4edb6a"]
Sep 09 12:33:18.784641 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:18.784613    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-4edb6a"]
Sep 09 12:33:19.177182 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:19.177153    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7684-4edb6a" container="test-curl-7684-4edb6a"
Sep 09 12:33:20.183519 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:20.183443    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7684-4edb6a" podUID="642e1d38-1ad2-4c81-beb6-f2c98bcd9b29" containerName="test-curl-7684-4edb6a" containerID="cri-o://81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f" gracePeriod=2
Sep 09 12:33:20.185141 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:20.185109    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-4edb6a\" is forbidden: User \"system:node:ip-10-0-133-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-152.ec2.internal' and this object" podUID="642e1d38-1ad2-4c81-beb6-f2c98bcd9b29" pod="opendatahub/test-curl-7684-4edb6a"
Sep 09 12:33:20.362831 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:20.362804    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-d0b62a"
Sep 09 12:33:20.364583 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:20.364547    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-d0b62a\" is forbidden: User \"system:node:ip-10-0-133-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-152.ec2.internal' and this object" podUID="7398f5be-078b-41e5-ab2e-bbe53c096e41" pod="opendatahub/test-curl-7684-d0b62a"
Sep 09 12:33:20.494126 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:20.494047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4dd6\" (UniqueName: \"kubernetes.io/projected/7398f5be-078b-41e5-ab2e-bbe53c096e41-kube-api-access-t4dd6\") pod \"7398f5be-078b-41e5-ab2e-bbe53c096e41\" (UID: \"7398f5be-078b-41e5-ab2e-bbe53c096e41\") "
Sep 09 12:33:20.496334 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:20.496309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7398f5be-078b-41e5-ab2e-bbe53c096e41-kube-api-access-t4dd6" (OuterVolumeSpecName: "kube-api-access-t4dd6") pod "7398f5be-078b-41e5-ab2e-bbe53c096e41" (UID: "7398f5be-078b-41e5-ab2e-bbe53c096e41"). InnerVolumeSpecName "kube-api-access-t4dd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:33:20.595165 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:20.595129    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4dd6\" (UniqueName: \"kubernetes.io/projected/7398f5be-078b-41e5-ab2e-bbe53c096e41-kube-api-access-t4dd6\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:33:21.188362 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:21.188324    2578 generic.go:356] "Generic (PLEG): container finished" podID="7398f5be-078b-41e5-ab2e-bbe53c096e41" containerID="99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900" exitCode=137
Sep 09 12:33:21.188362 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:21.188363    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-d0b62a"
Sep 09 12:33:21.188809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:21.188427    2578 scope.go:117] "RemoveContainer" containerID="99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900"
Sep 09 12:33:21.190390 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:21.190365    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-d0b62a\" is forbidden: User \"system:node:ip-10-0-133-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-152.ec2.internal' and this object" podUID="7398f5be-078b-41e5-ab2e-bbe53c096e41" pod="opendatahub/test-curl-7684-d0b62a"
Sep 09 12:33:21.197741 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:21.197724    2578 scope.go:117] "RemoveContainer" containerID="99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900"
Sep 09 12:33:21.197834 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:21.197813    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-d0b62a\" is forbidden: User \"system:node:ip-10-0-133-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-152.ec2.internal' and this object" podUID="7398f5be-078b-41e5-ab2e-bbe53c096e41" pod="opendatahub/test-curl-7684-d0b62a"
Sep 09 12:33:21.198018 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:21.197995    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900\": container with ID starting with 99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900 not found: ID does not exist" containerID="99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900"
Sep 09 12:33:21.198081 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:21.198030    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900"} err="failed to get container status \"99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900\": rpc error: code = NotFound desc = could not find container \"99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900\": container with ID starting with 99ab4f5fa6e2c95b08b8b8002888bed32597cefde764a7c63a2341e3eccac900 not found: ID does not exist"
Sep 09 12:33:22.192439 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:22.192404    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7398f5be-078b-41e5-ab2e-bbe53c096e41" path="/var/lib/kubelet/pods/7398f5be-078b-41e5-ab2e-bbe53c096e41/volumes"
Sep 09 12:33:22.359767 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:22.359745    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-4edb6a"
Sep 09 12:33:22.408026 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:22.407993    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkctt\" (UniqueName: \"kubernetes.io/projected/642e1d38-1ad2-4c81-beb6-f2c98bcd9b29-kube-api-access-fkctt\") pod \"642e1d38-1ad2-4c81-beb6-f2c98bcd9b29\" (UID: \"642e1d38-1ad2-4c81-beb6-f2c98bcd9b29\") "
Sep 09 12:33:22.410232 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:22.410180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642e1d38-1ad2-4c81-beb6-f2c98bcd9b29-kube-api-access-fkctt" (OuterVolumeSpecName: "kube-api-access-fkctt") pod "642e1d38-1ad2-4c81-beb6-f2c98bcd9b29" (UID: "642e1d38-1ad2-4c81-beb6-f2c98bcd9b29"). InnerVolumeSpecName "kube-api-access-fkctt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:33:22.509505 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:22.509414    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkctt\" (UniqueName: \"kubernetes.io/projected/642e1d38-1ad2-4c81-beb6-f2c98bcd9b29-kube-api-access-fkctt\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:33:23.201511 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:23.201476    2578 generic.go:356] "Generic (PLEG): container finished" podID="642e1d38-1ad2-4c81-beb6-f2c98bcd9b29" containerID="81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f" exitCode=137
Sep 09 12:33:23.201511 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:23.201512    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-4edb6a"
Sep 09 12:33:23.202050 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:23.201581    2578 scope.go:117] "RemoveContainer" containerID="81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f"
Sep 09 12:33:23.210345 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:23.210320    2578 scope.go:117] "RemoveContainer" containerID="81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f"
Sep 09 12:33:23.210646 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:23.210629    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f\": container with ID starting with 81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f not found: ID does not exist" containerID="81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f"
Sep 09 12:33:23.210703 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:23.210655    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f"} err="failed to get container status \"81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f\": rpc error: code = NotFound desc = could not find container \"81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f\": container with ID starting with 81982d7b8ffe522c5367146478893f49bbaf41a242aeddee1b559395bdc22a9f not found: ID does not exist"
Sep 09 12:33:23.211705 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:23.211679    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-4edb6a\" is forbidden: User \"system:node:ip-10-0-133-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-152.ec2.internal' and this object" podUID="642e1d38-1ad2-4c81-beb6-f2c98bcd9b29" pod="opendatahub/test-curl-7684-4edb6a"
Sep 09 12:33:24.186724 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.186696    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="642e1d38-1ad2-4c81-beb6-f2c98bcd9b29" path="/var/lib/kubelet/pods/642e1d38-1ad2-4c81-beb6-f2c98bcd9b29/volumes"
Sep 09 12:33:24.760895 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.760863    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"]
Sep 09 12:33:24.761282 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.761246    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="642e1d38-1ad2-4c81-beb6-f2c98bcd9b29" containerName="test-curl-7684-4edb6a"
Sep 09 12:33:24.761282 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.761257    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="642e1d38-1ad2-4c81-beb6-f2c98bcd9b29" containerName="test-curl-7684-4edb6a"
Sep 09 12:33:24.761363 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.761323    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="642e1d38-1ad2-4c81-beb6-f2c98bcd9b29" containerName="test-curl-7684-4edb6a"
Sep 09 12:33:24.765907 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.765888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:33:24.767756 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.767741    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f09b7dee\""
Sep 09 12:33:24.774376 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.774358    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"]
Sep 09 12:33:24.797285 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.797259    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"]
Sep 09 12:33:24.800487 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.800472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:33:24.802372 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.802345    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f09b7dee-dockercfg-6lbzm\""
Sep 09 12:33:24.810537 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.810505    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"]
Sep 09 12:33:24.830038 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.830010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p7b4\" (UniqueName: \"kubernetes.io/projected/b3c856d7-6082-4eef-89fe-4462d02fff2b-kube-api-access-8p7b4\") pod \"payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p\" (UID: \"b3c856d7-6082-4eef-89fe-4462d02fff2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:33:24.830185 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.830043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99883966-e7c0-4785-9476-c634db9e2d6f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p\" (UID: \"99883966-e7c0-4785-9476-c634db9e2d6f\") " pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:33:24.830185 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.830074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7jn\" (UniqueName: \"kubernetes.io/projected/99883966-e7c0-4785-9476-c634db9e2d6f-kube-api-access-4n7jn\") pod \"payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p\" (UID: \"99883966-e7c0-4785-9476-c634db9e2d6f\") " pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:33:24.830185 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.830096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c856d7-6082-4eef-89fe-4462d02fff2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p\" (UID: \"b3c856d7-6082-4eef-89fe-4462d02fff2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:33:24.931469 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.931429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p7b4\" (UniqueName: \"kubernetes.io/projected/b3c856d7-6082-4eef-89fe-4462d02fff2b-kube-api-access-8p7b4\") pod \"payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p\" (UID: \"b3c856d7-6082-4eef-89fe-4462d02fff2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:33:24.931634 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.931481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99883966-e7c0-4785-9476-c634db9e2d6f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p\" (UID: \"99883966-e7c0-4785-9476-c634db9e2d6f\") " pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:33:24.931634 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.931530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7jn\" (UniqueName: \"kubernetes.io/projected/99883966-e7c0-4785-9476-c634db9e2d6f-kube-api-access-4n7jn\") pod \"payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p\" (UID: \"99883966-e7c0-4785-9476-c634db9e2d6f\") " pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:33:24.931634 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.931551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c856d7-6082-4eef-89fe-4462d02fff2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p\" (UID: \"b3c856d7-6082-4eef-89fe-4462d02fff2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:33:24.932286 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.932263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99883966-e7c0-4785-9476-c634db9e2d6f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p\" (UID: \"99883966-e7c0-4785-9476-c634db9e2d6f\") " pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:33:24.932382 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.932297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c856d7-6082-4eef-89fe-4462d02fff2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p\" (UID: \"b3c856d7-6082-4eef-89fe-4462d02fff2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:33:24.940067 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.940047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7jn\" (UniqueName: \"kubernetes.io/projected/99883966-e7c0-4785-9476-c634db9e2d6f-kube-api-access-4n7jn\") pod \"payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p\" (UID: \"99883966-e7c0-4785-9476-c634db9e2d6f\") " pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:33:24.940162 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:24.940131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p7b4\" (UniqueName: \"kubernetes.io/projected/b3c856d7-6082-4eef-89fe-4462d02fff2b-kube-api-access-8p7b4\") pod \"payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p\" (UID: \"b3c856d7-6082-4eef-89fe-4462d02fff2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:33:25.076512 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:25.076420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:33:25.112404 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:25.112373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:33:25.353944 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:25.353916    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"]
Sep 09 12:33:25.359471 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:33:25.359422    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c856d7_6082_4eef_89fe_4462d02fff2b.slice/crio-d7308361900fe73b0b079b1d8bbd7a24084629b3de2a7954669e41b3cf6f5a8a WatchSource:0}: Error finding container d7308361900fe73b0b079b1d8bbd7a24084629b3de2a7954669e41b3cf6f5a8a: Status 404 returned error can't find the container with id d7308361900fe73b0b079b1d8bbd7a24084629b3de2a7954669e41b3cf6f5a8a
Sep 09 12:33:25.397598 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:25.397573    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"]
Sep 09 12:33:25.400264 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:33:25.400232    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99883966_e7c0_4785_9476_c634db9e2d6f.slice/crio-8515abba5cb0e379c50a2c3f329372817cdc53ebf964740706be714f862b44d2 WatchSource:0}: Error finding container 8515abba5cb0e379c50a2c3f329372817cdc53ebf964740706be714f862b44d2: Status 404 returned error can't find the container with id 8515abba5cb0e379c50a2c3f329372817cdc53ebf964740706be714f862b44d2
Sep 09 12:33:26.222146 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:26.222063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p" event={"ID":"99883966-e7c0-4785-9476-c634db9e2d6f","Type":"ContainerStarted","Data":"8515abba5cb0e379c50a2c3f329372817cdc53ebf964740706be714f862b44d2"}
Sep 09 12:33:26.224558 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:26.224441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p" event={"ID":"b3c856d7-6082-4eef-89fe-4462d02fff2b","Type":"ContainerStarted","Data":"d7308361900fe73b0b079b1d8bbd7a24084629b3de2a7954669e41b3cf6f5a8a"}
Sep 09 12:33:28.137254 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:28.137145    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:33:28.238175 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:28.237854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p" event={"ID":"99883966-e7c0-4785-9476-c634db9e2d6f","Type":"ContainerStarted","Data":"172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d"}
Sep 09 12:33:28.238500 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:28.238422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:33:28.239865 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:28.239842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p" event={"ID":"b3c856d7-6082-4eef-89fe-4462d02fff2b","Type":"ContainerStarted","Data":"c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e"}
Sep 09 12:33:28.239977 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:28.239970    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:33:28.258466 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:28.258406    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p" podStartSLOduration=1.8016334440000001 podStartE2EDuration="4.25838879s" totalImagesPullingTime="2.456755346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:33:28.254668618 +0000 UTC m=+828.598266591" watchObservedRunningTime="2026-09-09 12:33:28.25838879 +0000 UTC m=+828.601986739"
Sep 09 12:33:28.274561 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:28.274501    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p" podStartSLOduration=1.7700942240000002 podStartE2EDuration="4.274481228s" totalImagesPullingTime="2.504387004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:33:28.274322502 +0000 UTC m=+828.617920451" watchObservedRunningTime="2026-09-09 12:33:28.274481228 +0000 UTC m=+828.618079178"
Sep 09 12:33:32.116288 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.116253    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"]
Sep 09 12:33:32.120726 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.120710    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.122584 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.122566    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-c356d768-kserve-self-signed-certs\""
Sep 09 12:33:32.132268 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.132248    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"]
Sep 09 12:33:32.203392 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.203359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-tmp-dir\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.203392 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.203398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-model-cache\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.203600 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.203430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-dshm\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.203600 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.203527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a270143-a64f-402b-bc8b-cfdd357cb616-tls-certs\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.203600 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.203576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jvb\" (UniqueName: \"kubernetes.io/projected/8a270143-a64f-402b-bc8b-cfdd357cb616-kube-api-access-g7jvb\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.203696 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.203628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-home\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.304404 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.304371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-tmp-dir\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.304583 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.304415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-model-cache\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.304583 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.304448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-dshm\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.304583 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.304484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a270143-a64f-402b-bc8b-cfdd357cb616-tls-certs\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.304583 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.304529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jvb\" (UniqueName: \"kubernetes.io/projected/8a270143-a64f-402b-bc8b-cfdd357cb616-kube-api-access-g7jvb\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.304583 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.304575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-home\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.304873 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.304856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-tmp-dir\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.305165 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.305144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-home\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.305303 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.305245    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-model-cache\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.307057 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.307034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-dshm\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.307503 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.307475    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a270143-a64f-402b-bc8b-cfdd357cb616-tls-certs\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.313268 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.313239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jvb\" (UniqueName: \"kubernetes.io/projected/8a270143-a64f-402b-bc8b-cfdd357cb616-kube-api-access-g7jvb\") pod \"e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") " pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.432000 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.431949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:32.704780 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:32.704753    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"]
Sep 09 12:33:32.706550 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:33:32.706509    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a270143_a64f_402b_bc8b_cfdd357cb616.slice/crio-12f69640cfac542e388f249c1dd325730de67d2c427862be01ef4c1a909b4871 WatchSource:0}: Error finding container 12f69640cfac542e388f249c1dd325730de67d2c427862be01ef4c1a909b4871: Status 404 returned error can't find the container with id 12f69640cfac542e388f249c1dd325730de67d2c427862be01ef4c1a909b4871
Sep 09 12:33:33.272669 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:33.272637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv" event={"ID":"8a270143-a64f-402b-bc8b-cfdd357cb616","Type":"ContainerStarted","Data":"7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c"}
Sep 09 12:33:33.272669 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:33.272678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv" event={"ID":"8a270143-a64f-402b-bc8b-cfdd357cb616","Type":"ContainerStarted","Data":"12f69640cfac542e388f249c1dd325730de67d2c427862be01ef4c1a909b4871"}
Sep 09 12:33:33.292334 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:33.292286    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv" podStartSLOduration=1.292270171 podStartE2EDuration="1.292270171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:33:33.290341577 +0000 UTC m=+833.633939528" watchObservedRunningTime="2026-09-09 12:33:33.292270171 +0000 UTC m=+833.635868119"
Sep 09 12:33:39.247428 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:39.247395    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:33:39.247841 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:39.247468    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:33:42.432633 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:42.432602    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:42.433046 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:42.432650    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:42.445565 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:42.445538    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:43.332183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.332147    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"]
Sep 09 12:33:43.337867 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.337839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:43.339929 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.339900    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5e5923-serving-cert\""
Sep 09 12:33:43.340069 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.339959    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 12:33:43.340694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.340646    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 12:33:43.340694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.340677    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 12:33:43.340875 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.340719    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5rdxz\""
Sep 09 12:33:43.343397 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.343377    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:33:43.345065 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.345043    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"]
Sep 09 12:33:43.420248 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.420177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-metrics-certs\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:43.420388 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.420301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:43.420388 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.420333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm9zt\" (UniqueName: \"kubernetes.io/projected/c99afdec-1abf-47e8-8046-c7bbc9db37c7-kube-api-access-sm9zt\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:43.420500 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.420423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99afdec-1abf-47e8-8046-c7bbc9db37c7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:43.521197 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.521156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99afdec-1abf-47e8-8046-c7bbc9db37c7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:43.521674 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.521279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-metrics-certs\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:43.521674 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.521321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:43.521674 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.521347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9zt\" (UniqueName: \"kubernetes.io/projected/c99afdec-1abf-47e8-8046-c7bbc9db37c7-kube-api-access-sm9zt\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:43.521674 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:43.521443    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-serving-cert: secret "maas-api-e2e-inf-b-5e5923-serving-cert" not found
Sep 09 12:33:43.521674 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:43.521511    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls podName:c99afdec-1abf-47e8-8046-c7bbc9db37c7 nodeName:}" failed. No retries permitted until 2026-09-09 12:33:44.02149463 +0000 UTC m=+844.365092556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls") pod "maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2" (UID: "c99afdec-1abf-47e8-8046-c7bbc9db37c7") : secret "maas-api-e2e-inf-b-5e5923-serving-cert" not found
Sep 09 12:33:43.521904 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.521837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99afdec-1abf-47e8-8046-c7bbc9db37c7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:43.523926 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.523902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-metrics-certs\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:43.529812 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:43.529792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9zt\" (UniqueName: \"kubernetes.io/projected/c99afdec-1abf-47e8-8046-c7bbc9db37c7-kube-api-access-sm9zt\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:44.026299 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:44.026258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:44.026595 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:44.026322    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-serving-cert: secret "maas-api-e2e-inf-b-5e5923-serving-cert" not found
Sep 09 12:33:44.026595 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:33:44.026415    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls podName:c99afdec-1abf-47e8-8046-c7bbc9db37c7 nodeName:}" failed. No retries permitted until 2026-09-09 12:33:45.026400466 +0000 UTC m=+845.369998391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls") pod "maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2" (UID: "c99afdec-1abf-47e8-8046-c7bbc9db37c7") : secret "maas-api-e2e-inf-b-5e5923-serving-cert" not found
Sep 09 12:33:45.036637 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:45.036605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:45.039047 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:45.039027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls\") pod \"maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:45.152472 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:45.152434    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:45.430638 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:45.430582    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"]
Sep 09 12:33:45.432647 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:33:45.432621    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99afdec_1abf_47e8_8046_c7bbc9db37c7.slice/crio-7664997f203006f25b34cb93174ff121a8a2dcbc5862334289038d4bb38f949e WatchSource:0}: Error finding container 7664997f203006f25b34cb93174ff121a8a2dcbc5862334289038d4bb38f949e: Status 404 returned error can't find the container with id 7664997f203006f25b34cb93174ff121a8a2dcbc5862334289038d4bb38f949e
Sep 09 12:33:45.944695 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:45.944657    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:33:46.345472 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:46.345391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2" event={"ID":"c99afdec-1abf-47e8-8046-c7bbc9db37c7","Type":"ContainerStarted","Data":"7664997f203006f25b34cb93174ff121a8a2dcbc5862334289038d4bb38f949e"}
Sep 09 12:33:48.361344 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:48.361192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2" event={"ID":"c99afdec-1abf-47e8-8046-c7bbc9db37c7","Type":"ContainerStarted","Data":"5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a"}
Sep 09 12:33:48.361344 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:48.361311    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:33:48.378102 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:48.378053    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2" podStartSLOduration=2.8186701579999998 podStartE2EDuration="5.378038936s" totalImagesPullingTime="2.559368778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:33:48.376735604 +0000 UTC m=+848.720333553" watchObservedRunningTime="2026-09-09 12:33:48.378038936 +0000 UTC m=+848.721636884"
Sep 09 12:33:54.372626 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:33:54.372595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:34:04.792407 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:04.792370    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"]
Sep 09 12:34:04.792770 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:04.792664    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p" podUID="99883966-e7c0-4785-9476-c634db9e2d6f" containerName="payload-processing" containerID="cri-o://172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d" gracePeriod=30
Sep 09 12:34:04.809066 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:04.808899    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"]
Sep 09 12:34:04.810933 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:04.809875    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p" podUID="b3c856d7-6082-4eef-89fe-4462d02fff2b" containerName="payload-pre-processing" containerID="cri-o://c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e" gracePeriod=30
Sep 09 12:34:05.041458 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.041426    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:34:05.111200 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.111172    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:34:05.114701 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.114669    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:34:05.130817 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.130794    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p7b4\" (UniqueName: \"kubernetes.io/projected/b3c856d7-6082-4eef-89fe-4462d02fff2b-kube-api-access-8p7b4\") pod \"b3c856d7-6082-4eef-89fe-4462d02fff2b\" (UID: \"b3c856d7-6082-4eef-89fe-4462d02fff2b\") "
Sep 09 12:34:05.130947 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.130831    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99883966-e7c0-4785-9476-c634db9e2d6f-plugins-config-volume\") pod \"99883966-e7c0-4785-9476-c634db9e2d6f\" (UID: \"99883966-e7c0-4785-9476-c634db9e2d6f\") "
Sep 09 12:34:05.130947 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.130914    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n7jn\" (UniqueName: \"kubernetes.io/projected/99883966-e7c0-4785-9476-c634db9e2d6f-kube-api-access-4n7jn\") pod \"99883966-e7c0-4785-9476-c634db9e2d6f\" (UID: \"99883966-e7c0-4785-9476-c634db9e2d6f\") "
Sep 09 12:34:05.130947 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.130931    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c856d7-6082-4eef-89fe-4462d02fff2b-plugins-config-volume\") pod \"b3c856d7-6082-4eef-89fe-4462d02fff2b\" (UID: \"b3c856d7-6082-4eef-89fe-4462d02fff2b\") "
Sep 09 12:34:05.131319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.131287    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99883966-e7c0-4785-9476-c634db9e2d6f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99883966-e7c0-4785-9476-c634db9e2d6f" (UID: "99883966-e7c0-4785-9476-c634db9e2d6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:34:05.131450 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.131396    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c856d7-6082-4eef-89fe-4462d02fff2b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3c856d7-6082-4eef-89fe-4462d02fff2b" (UID: "b3c856d7-6082-4eef-89fe-4462d02fff2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:34:05.133191 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.133164    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c856d7-6082-4eef-89fe-4462d02fff2b-kube-api-access-8p7b4" (OuterVolumeSpecName: "kube-api-access-8p7b4") pod "b3c856d7-6082-4eef-89fe-4462d02fff2b" (UID: "b3c856d7-6082-4eef-89fe-4462d02fff2b"). InnerVolumeSpecName "kube-api-access-8p7b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:34:05.133191 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.133185    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99883966-e7c0-4785-9476-c634db9e2d6f-kube-api-access-4n7jn" (OuterVolumeSpecName: "kube-api-access-4n7jn") pod "99883966-e7c0-4785-9476-c634db9e2d6f" (UID: "99883966-e7c0-4785-9476-c634db9e2d6f"). InnerVolumeSpecName "kube-api-access-4n7jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:34:05.232113 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.232078    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n7jn\" (UniqueName: \"kubernetes.io/projected/99883966-e7c0-4785-9476-c634db9e2d6f-kube-api-access-4n7jn\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:05.232113 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.232112    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c856d7-6082-4eef-89fe-4462d02fff2b-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:05.232418 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.232126    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p7b4\" (UniqueName: \"kubernetes.io/projected/b3c856d7-6082-4eef-89fe-4462d02fff2b-kube-api-access-8p7b4\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:05.232418 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.232134    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99883966-e7c0-4785-9476-c634db9e2d6f-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:05.448924 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.448886    2578 generic.go:356] "Generic (PLEG): container finished" podID="99883966-e7c0-4785-9476-c634db9e2d6f" containerID="172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d" exitCode=0
Sep 09 12:34:05.449096 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.448930    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"
Sep 09 12:34:05.449096 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.448970    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p" event={"ID":"99883966-e7c0-4785-9476-c634db9e2d6f","Type":"ContainerDied","Data":"172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d"}
Sep 09 12:34:05.449096 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.449019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p" event={"ID":"99883966-e7c0-4785-9476-c634db9e2d6f","Type":"ContainerDied","Data":"8515abba5cb0e379c50a2c3f329372817cdc53ebf964740706be714f862b44d2"}
Sep 09 12:34:05.449096 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.449047    2578 scope.go:117] "RemoveContainer" containerID="172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d"
Sep 09 12:34:05.450405 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.450380    2578 generic.go:356] "Generic (PLEG): container finished" podID="b3c856d7-6082-4eef-89fe-4462d02fff2b" containerID="c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e" exitCode=0
Sep 09 12:34:05.450517 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.450400    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p" event={"ID":"b3c856d7-6082-4eef-89fe-4462d02fff2b","Type":"ContainerDied","Data":"c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e"}
Sep 09 12:34:05.450517 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.450446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p" event={"ID":"b3c856d7-6082-4eef-89fe-4462d02fff2b","Type":"ContainerDied","Data":"d7308361900fe73b0b079b1d8bbd7a24084629b3de2a7954669e41b3cf6f5a8a"}
Sep 09 12:34:05.450517 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.450462    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"
Sep 09 12:34:05.459044 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.458996    2578 scope.go:117] "RemoveContainer" containerID="172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d"
Sep 09 12:34:05.459390 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:34:05.459371    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d\": container with ID starting with 172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d not found: ID does not exist" containerID="172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d"
Sep 09 12:34:05.459470 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.459397    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d"} err="failed to get container status \"172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d\": rpc error: code = NotFound desc = could not find container \"172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d\": container with ID starting with 172ae1f812e9cdb9c008067f29a9eecc23cb6017456a1b979d89dd61b9decc2d not found: ID does not exist"
Sep 09 12:34:05.459470 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.459412    2578 scope.go:117] "RemoveContainer" containerID="c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e"
Sep 09 12:34:05.467651 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.467636    2578 scope.go:117] "RemoveContainer" containerID="c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e"
Sep 09 12:34:05.467906 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:34:05.467888    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e\": container with ID starting with c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e not found: ID does not exist" containerID="c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e"
Sep 09 12:34:05.467950 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.467913    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e"} err="failed to get container status \"c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e\": rpc error: code = NotFound desc = could not find container \"c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e\": container with ID starting with c4bfe5af4848512d824b01b335ae87628eccbdcd135e940f3814e95edf64f35e not found: ID does not exist"
Sep 09 12:34:05.474869 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.474848    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"]
Sep 09 12:34:05.478982 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.478965    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f09b7dee-7b8895b69d-hv98p"]
Sep 09 12:34:05.489871 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.489850    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"]
Sep 09 12:34:05.493773 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:05.493752    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f09b7dee-7c58b9655-5746p"]
Sep 09 12:34:06.187054 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:06.187021    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99883966-e7c0-4785-9476-c634db9e2d6f" path="/var/lib/kubelet/pods/99883966-e7c0-4785-9476-c634db9e2d6f/volumes"
Sep 09 12:34:06.187434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:06.187372    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3c856d7-6082-4eef-89fe-4462d02fff2b" path="/var/lib/kubelet/pods/b3c856d7-6082-4eef-89fe-4462d02fff2b/volumes"
Sep 09 12:34:09.174388 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.174356    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"]
Sep 09 12:34:09.175592 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.175564    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c856d7-6082-4eef-89fe-4462d02fff2b" containerName="payload-pre-processing"
Sep 09 12:34:09.175592 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.175587    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c856d7-6082-4eef-89fe-4462d02fff2b" containerName="payload-pre-processing"
Sep 09 12:34:09.175782 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.175626    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99883966-e7c0-4785-9476-c634db9e2d6f" containerName="payload-processing"
Sep 09 12:34:09.175782 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.175636    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99883966-e7c0-4785-9476-c634db9e2d6f" containerName="payload-processing"
Sep 09 12:34:09.175782 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.175730    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3c856d7-6082-4eef-89fe-4462d02fff2b" containerName="payload-pre-processing"
Sep 09 12:34:09.175782 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.175746    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99883966-e7c0-4785-9476-c634db9e2d6f" containerName="payload-processing"
Sep 09 12:34:09.178862 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.178840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.180797 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.180772    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-dbdca9a8-serving-cert\""
Sep 09 12:34:09.191331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.191307    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"]
Sep 09 12:34:09.269044 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.269007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-maas-api-tls\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.269238 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.269058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-metrics-certs\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.269238 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.269076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflwp\" (UniqueName: \"kubernetes.io/projected/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-kube-api-access-nflwp\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.269238 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.269140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-openshift-service-ca\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.370130 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.370087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-maas-api-tls\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.370319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.370163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-metrics-certs\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.370319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.370189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflwp\" (UniqueName: \"kubernetes.io/projected/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-kube-api-access-nflwp\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.370461 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.370340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-openshift-service-ca\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.371077 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.371050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-openshift-service-ca\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.372812 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.372792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-metrics-certs\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.372942 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.372925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-maas-api-tls\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.378790 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.378764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflwp\" (UniqueName: \"kubernetes.io/projected/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-kube-api-access-nflwp\") pod \"maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.494402 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.494318    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:09.767336 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.767318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"]
Sep 09 12:34:09.769714 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:34:09.769678    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4359a90f_0dfd_46cd_8bf8_2c8160df83c3.slice/crio-e557e5b03efe7f207b443f8cedb8fbfc22f214d466672f9cae24a53328d089ae WatchSource:0}: Error finding container e557e5b03efe7f207b443f8cedb8fbfc22f214d466672f9cae24a53328d089ae: Status 404 returned error can't find the container with id e557e5b03efe7f207b443f8cedb8fbfc22f214d466672f9cae24a53328d089ae
Sep 09 12:34:09.992079 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.992042    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"]
Sep 09 12:34:09.995031 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.995016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:09.997274 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.997253    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-dbdca9a8-dockercfg-jczdl\""
Sep 09 12:34:09.997367 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:09.997289    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-dbdca9a8\""
Sep 09 12:34:10.005660 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.005636    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"]
Sep 09 12:34:10.054598 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.054573    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"]
Sep 09 12:34:10.057492 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.057477    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:10.066632 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.066604    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"]
Sep 09 12:34:10.077528 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.077488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff803f24-ae09-407f-b7c0-be2b6c92eed0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n\" (UID: \"ff803f24-ae09-407f-b7c0-be2b6c92eed0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:10.077668 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.077568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffec367-cdea-4176-adb9-4edf653641be-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72\" (UID: \"9ffec367-cdea-4176-adb9-4edf653641be\") " pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:10.077718 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.077676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n7xn\" (UniqueName: \"kubernetes.io/projected/ff803f24-ae09-407f-b7c0-be2b6c92eed0-kube-api-access-9n7xn\") pod \"payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n\" (UID: \"ff803f24-ae09-407f-b7c0-be2b6c92eed0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:10.077718 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.077700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tdgn\" (UniqueName: \"kubernetes.io/projected/9ffec367-cdea-4176-adb9-4edf653641be-kube-api-access-6tdgn\") pod \"payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72\" (UID: \"9ffec367-cdea-4176-adb9-4edf653641be\") " pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:10.178616 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.178588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n7xn\" (UniqueName: \"kubernetes.io/projected/ff803f24-ae09-407f-b7c0-be2b6c92eed0-kube-api-access-9n7xn\") pod \"payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n\" (UID: \"ff803f24-ae09-407f-b7c0-be2b6c92eed0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:10.179013 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.178627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tdgn\" (UniqueName: \"kubernetes.io/projected/9ffec367-cdea-4176-adb9-4edf653641be-kube-api-access-6tdgn\") pod \"payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72\" (UID: \"9ffec367-cdea-4176-adb9-4edf653641be\") " pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:10.179013 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.178715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff803f24-ae09-407f-b7c0-be2b6c92eed0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n\" (UID: \"ff803f24-ae09-407f-b7c0-be2b6c92eed0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:10.179013 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.178775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffec367-cdea-4176-adb9-4edf653641be-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72\" (UID: \"9ffec367-cdea-4176-adb9-4edf653641be\") " pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:10.179457 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.179436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff803f24-ae09-407f-b7c0-be2b6c92eed0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n\" (UID: \"ff803f24-ae09-407f-b7c0-be2b6c92eed0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:10.179561 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.179540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffec367-cdea-4176-adb9-4edf653641be-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72\" (UID: \"9ffec367-cdea-4176-adb9-4edf653641be\") " pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:10.187019 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.186994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tdgn\" (UniqueName: \"kubernetes.io/projected/9ffec367-cdea-4176-adb9-4edf653641be-kube-api-access-6tdgn\") pod \"payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72\" (UID: \"9ffec367-cdea-4176-adb9-4edf653641be\") " pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:10.187149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.187026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n7xn\" (UniqueName: \"kubernetes.io/projected/ff803f24-ae09-407f-b7c0-be2b6c92eed0-kube-api-access-9n7xn\") pod \"payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n\" (UID: \"ff803f24-ae09-407f-b7c0-be2b6c92eed0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:10.306159 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.306070    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:10.448042 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.448007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:10.479588 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.479549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff" event={"ID":"4359a90f-0dfd-46cd-8bf8-2c8160df83c3","Type":"ContainerStarted","Data":"06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414"}
Sep 09 12:34:10.479755 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.479600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff" event={"ID":"4359a90f-0dfd-46cd-8bf8-2c8160df83c3","Type":"ContainerStarted","Data":"e557e5b03efe7f207b443f8cedb8fbfc22f214d466672f9cae24a53328d089ae"}
Sep 09 12:34:10.479755 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.479734    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:10.497615 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.497556    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff" podStartSLOduration=1.225142151 podStartE2EDuration="1.497533524s" totalImagesPullingTime="272.391373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:34:10.496364623 +0000 UTC m=+870.839962574" watchObservedRunningTime="2026-09-09 12:34:10.497533524 +0000 UTC m=+870.841131473"
Sep 09 12:34:10.580148 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.580122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"]
Sep 09 12:34:10.581753 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:34:10.581722    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffec367_cdea_4176_adb9_4edf653641be.slice/crio-bb474f630c582541512710856560d7ca7b9f26f4681ef3a711ee548d56f4ebc9 WatchSource:0}: Error finding container bb474f630c582541512710856560d7ca7b9f26f4681ef3a711ee548d56f4ebc9: Status 404 returned error can't find the container with id bb474f630c582541512710856560d7ca7b9f26f4681ef3a711ee548d56f4ebc9
Sep 09 12:34:10.719385 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:10.719353    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"]
Sep 09 12:34:10.722310 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:34:10.722283    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff803f24_ae09_407f_b7c0_be2b6c92eed0.slice/crio-0ba80d31ef710063a08d2f8f384305f0021f41e22ad70e8c1c0bdad6b5433dba WatchSource:0}: Error finding container 0ba80d31ef710063a08d2f8f384305f0021f41e22ad70e8c1c0bdad6b5433dba: Status 404 returned error can't find the container with id 0ba80d31ef710063a08d2f8f384305f0021f41e22ad70e8c1c0bdad6b5433dba
Sep 09 12:34:11.486463 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:11.486418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n" event={"ID":"ff803f24-ae09-407f-b7c0-be2b6c92eed0","Type":"ContainerStarted","Data":"c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879"}
Sep 09 12:34:11.486947 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:11.486489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n" event={"ID":"ff803f24-ae09-407f-b7c0-be2b6c92eed0","Type":"ContainerStarted","Data":"0ba80d31ef710063a08d2f8f384305f0021f41e22ad70e8c1c0bdad6b5433dba"}
Sep 09 12:34:11.486947 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:11.486553    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:11.488187 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:11.488155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72" event={"ID":"9ffec367-cdea-4176-adb9-4edf653641be","Type":"ContainerStarted","Data":"d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646"}
Sep 09 12:34:11.488325 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:11.488233    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72" event={"ID":"9ffec367-cdea-4176-adb9-4edf653641be","Type":"ContainerStarted","Data":"bb474f630c582541512710856560d7ca7b9f26f4681ef3a711ee548d56f4ebc9"}
Sep 09 12:34:11.488325 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:11.488310    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:11.502995 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:11.502946    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n" podStartSLOduration=1.502930971 podStartE2EDuration="1.502930971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:34:11.500705025 +0000 UTC m=+871.844302972" watchObservedRunningTime="2026-09-09 12:34:11.502930971 +0000 UTC m=+871.846528918"
Sep 09 12:34:11.516193 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:11.516120    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72" podStartSLOduration=2.516105261 podStartE2EDuration="2.516105261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:34:11.516029488 +0000 UTC m=+871.859627440" watchObservedRunningTime="2026-09-09 12:34:11.516105261 +0000 UTC m=+871.859703208"
Sep 09 12:34:16.494019 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:16.493985    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:22.494838 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:22.494808    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:22.495335 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:22.495205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:28.329421 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:28.329384    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:34:39.811279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:39.811164    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"]
Sep 09 12:34:39.811732 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:39.811436    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff" podUID="4359a90f-0dfd-46cd-8bf8-2c8160df83c3" containerName="maas-api" containerID="cri-o://06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414" gracePeriod=30
Sep 09 12:34:39.876024 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:39.875965    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"]
Sep 09 12:34:39.876312 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:39.876256    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72" podUID="9ffec367-cdea-4176-adb9-4edf653641be" containerName="payload-processing" containerID="cri-o://d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646" gracePeriod=30
Sep 09 12:34:39.884359 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:39.884123    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"]
Sep 09 12:34:39.884482 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:39.884363    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n" podUID="ff803f24-ae09-407f-b7c0-be2b6c92eed0" containerName="payload-pre-processing" containerID="cri-o://c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879" gracePeriod=30
Sep 09 12:34:40.179612 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.179436    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:34:40.179953 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.179934    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:34:40.192163 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.192138    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:40.196546 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.196527    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:40.200145 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.200129    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:40.271850 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.271816    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-metrics-certs\") pod \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") "
Sep 09 12:34:40.271850 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.271855    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tdgn\" (UniqueName: \"kubernetes.io/projected/9ffec367-cdea-4176-adb9-4edf653641be-kube-api-access-6tdgn\") pod \"9ffec367-cdea-4176-adb9-4edf653641be\" (UID: \"9ffec367-cdea-4176-adb9-4edf653641be\") "
Sep 09 12:34:40.272101 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.271881    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n7xn\" (UniqueName: \"kubernetes.io/projected/ff803f24-ae09-407f-b7c0-be2b6c92eed0-kube-api-access-9n7xn\") pod \"ff803f24-ae09-407f-b7c0-be2b6c92eed0\" (UID: \"ff803f24-ae09-407f-b7c0-be2b6c92eed0\") "
Sep 09 12:34:40.272101 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.271905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-openshift-service-ca\") pod \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") "
Sep 09 12:34:40.272101 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.271929    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nflwp\" (UniqueName: \"kubernetes.io/projected/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-kube-api-access-nflwp\") pod \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") "
Sep 09 12:34:40.272101 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.271975    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff803f24-ae09-407f-b7c0-be2b6c92eed0-plugins-config-volume\") pod \"ff803f24-ae09-407f-b7c0-be2b6c92eed0\" (UID: \"ff803f24-ae09-407f-b7c0-be2b6c92eed0\") "
Sep 09 12:34:40.272101 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.271999    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-maas-api-tls\") pod \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\" (UID: \"4359a90f-0dfd-46cd-8bf8-2c8160df83c3\") "
Sep 09 12:34:40.272101 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.272089    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffec367-cdea-4176-adb9-4edf653641be-plugins-config-volume\") pod \"9ffec367-cdea-4176-adb9-4edf653641be\" (UID: \"9ffec367-cdea-4176-adb9-4edf653641be\") "
Sep 09 12:34:40.272597 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.272569    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4359a90f-0dfd-46cd-8bf8-2c8160df83c3" (UID: "4359a90f-0dfd-46cd-8bf8-2c8160df83c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:34:40.273320 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.273140    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff803f24-ae09-407f-b7c0-be2b6c92eed0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff803f24-ae09-407f-b7c0-be2b6c92eed0" (UID: "ff803f24-ae09-407f-b7c0-be2b6c92eed0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:34:40.273803 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.273779    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffec367-cdea-4176-adb9-4edf653641be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ffec367-cdea-4176-adb9-4edf653641be" (UID: "9ffec367-cdea-4176-adb9-4edf653641be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:34:40.274931 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.274906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffec367-cdea-4176-adb9-4edf653641be-kube-api-access-6tdgn" (OuterVolumeSpecName: "kube-api-access-6tdgn") pod "9ffec367-cdea-4176-adb9-4edf653641be" (UID: "9ffec367-cdea-4176-adb9-4edf653641be"). InnerVolumeSpecName "kube-api-access-6tdgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:34:40.275029 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.274991    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-kube-api-access-nflwp" (OuterVolumeSpecName: "kube-api-access-nflwp") pod "4359a90f-0dfd-46cd-8bf8-2c8160df83c3" (UID: "4359a90f-0dfd-46cd-8bf8-2c8160df83c3"). InnerVolumeSpecName "kube-api-access-nflwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:34:40.276107 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.276073    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4359a90f-0dfd-46cd-8bf8-2c8160df83c3" (UID: "4359a90f-0dfd-46cd-8bf8-2c8160df83c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:34:40.276107 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.276092    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff803f24-ae09-407f-b7c0-be2b6c92eed0-kube-api-access-9n7xn" (OuterVolumeSpecName: "kube-api-access-9n7xn") pod "ff803f24-ae09-407f-b7c0-be2b6c92eed0" (UID: "ff803f24-ae09-407f-b7c0-be2b6c92eed0"). InnerVolumeSpecName "kube-api-access-9n7xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:34:40.276304 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.276135    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4359a90f-0dfd-46cd-8bf8-2c8160df83c3" (UID: "4359a90f-0dfd-46cd-8bf8-2c8160df83c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:34:40.373694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.373622    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-openshift-service-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:40.373694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.373646    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nflwp\" (UniqueName: \"kubernetes.io/projected/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-kube-api-access-nflwp\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:40.373694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.373657    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff803f24-ae09-407f-b7c0-be2b6c92eed0-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:40.373694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.373667    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-maas-api-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:40.373694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.373676    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffec367-cdea-4176-adb9-4edf653641be-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:40.373694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.373683    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4359a90f-0dfd-46cd-8bf8-2c8160df83c3-metrics-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:40.373694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.373692    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tdgn\" (UniqueName: \"kubernetes.io/projected/9ffec367-cdea-4176-adb9-4edf653641be-kube-api-access-6tdgn\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:40.373694 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.373699    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n7xn\" (UniqueName: \"kubernetes.io/projected/ff803f24-ae09-407f-b7c0-be2b6c92eed0-kube-api-access-9n7xn\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:34:40.637709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.637624    2578 generic.go:356] "Generic (PLEG): container finished" podID="4359a90f-0dfd-46cd-8bf8-2c8160df83c3" containerID="06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414" exitCode=0
Sep 09 12:34:40.637709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.637668    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"
Sep 09 12:34:40.637709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.637699    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff" event={"ID":"4359a90f-0dfd-46cd-8bf8-2c8160df83c3","Type":"ContainerDied","Data":"06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414"}
Sep 09 12:34:40.637958 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.637754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff" event={"ID":"4359a90f-0dfd-46cd-8bf8-2c8160df83c3","Type":"ContainerDied","Data":"e557e5b03efe7f207b443f8cedb8fbfc22f214d466672f9cae24a53328d089ae"}
Sep 09 12:34:40.637958 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.637785    2578 scope.go:117] "RemoveContainer" containerID="06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414"
Sep 09 12:34:40.639069 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.639038    2578 generic.go:356] "Generic (PLEG): container finished" podID="ff803f24-ae09-407f-b7c0-be2b6c92eed0" containerID="c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879" exitCode=0
Sep 09 12:34:40.639157 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.639077    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"
Sep 09 12:34:40.639157 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.639116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n" event={"ID":"ff803f24-ae09-407f-b7c0-be2b6c92eed0","Type":"ContainerDied","Data":"c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879"}
Sep 09 12:34:40.639296 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.639172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n" event={"ID":"ff803f24-ae09-407f-b7c0-be2b6c92eed0","Type":"ContainerDied","Data":"0ba80d31ef710063a08d2f8f384305f0021f41e22ad70e8c1c0bdad6b5433dba"}
Sep 09 12:34:40.640787 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.640752    2578 generic.go:356] "Generic (PLEG): container finished" podID="9ffec367-cdea-4176-adb9-4edf653641be" containerID="d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646" exitCode=0
Sep 09 12:34:40.640902 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.640795    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"
Sep 09 12:34:40.640902 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.640812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72" event={"ID":"9ffec367-cdea-4176-adb9-4edf653641be","Type":"ContainerDied","Data":"d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646"}
Sep 09 12:34:40.640902 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.640860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72" event={"ID":"9ffec367-cdea-4176-adb9-4edf653641be","Type":"ContainerDied","Data":"bb474f630c582541512710856560d7ca7b9f26f4681ef3a711ee548d56f4ebc9"}
Sep 09 12:34:40.648067 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.648052    2578 scope.go:117] "RemoveContainer" containerID="06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414"
Sep 09 12:34:40.648387 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:34:40.648365    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414\": container with ID starting with 06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414 not found: ID does not exist" containerID="06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414"
Sep 09 12:34:40.648490 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.648394    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414"} err="failed to get container status \"06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414\": rpc error: code = NotFound desc = could not find container \"06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414\": container with ID starting with 06b02f14f045174aeec46e227051506737a31d28c542b0d2add7991db792a414 not found: ID does not exist"
Sep 09 12:34:40.648490 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.648410    2578 scope.go:117] "RemoveContainer" containerID="c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879"
Sep 09 12:34:40.656566 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.656548    2578 scope.go:117] "RemoveContainer" containerID="c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879"
Sep 09 12:34:40.656789 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:34:40.656771    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879\": container with ID starting with c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879 not found: ID does not exist" containerID="c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879"
Sep 09 12:34:40.656840 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.656795    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879"} err="failed to get container status \"c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879\": rpc error: code = NotFound desc = could not find container \"c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879\": container with ID starting with c288ee00473683dacbaa1be8560a19490ebff25f82cb97e4d198458b90984879 not found: ID does not exist"
Sep 09 12:34:40.656840 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.656810    2578 scope.go:117] "RemoveContainer" containerID="d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646"
Sep 09 12:34:40.664144 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.664131    2578 scope.go:117] "RemoveContainer" containerID="d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646"
Sep 09 12:34:40.664423 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:34:40.664408    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646\": container with ID starting with d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646 not found: ID does not exist" containerID="d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646"
Sep 09 12:34:40.664480 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.664428    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646"} err="failed to get container status \"d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646\": rpc error: code = NotFound desc = could not find container \"d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646\": container with ID starting with d3c5ba3c75b68822647c22c143954142532bfb93bfca15c78050fd916b4bf646 not found: ID does not exist"
Sep 09 12:34:40.672449 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.672428    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"]
Sep 09 12:34:40.680397 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.680377    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dbdca9a8-65846f6b6-x6c5n"]
Sep 09 12:34:40.701841 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.701816    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"]
Sep 09 12:34:40.707498 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.707476    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dbdca9a8-6667dbc5f4-nqc72"]
Sep 09 12:34:40.727205 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.727170    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"]
Sep 09 12:34:40.733663 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:40.733641    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dbdca9a8-5dbd47454f-98mff"]
Sep 09 12:34:42.186035 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:42.186004    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4359a90f-0dfd-46cd-8bf8-2c8160df83c3" path="/var/lib/kubelet/pods/4359a90f-0dfd-46cd-8bf8-2c8160df83c3/volumes"
Sep 09 12:34:42.186460 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:42.186411    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ffec367-cdea-4176-adb9-4edf653641be" path="/var/lib/kubelet/pods/9ffec367-cdea-4176-adb9-4edf653641be/volumes"
Sep 09 12:34:42.186723 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:42.186711    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff803f24-ae09-407f-b7c0-be2b6c92eed0" path="/var/lib/kubelet/pods/ff803f24-ae09-407f-b7c0-be2b6c92eed0/volumes"
Sep 09 12:34:44.178718 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.178685    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"]
Sep 09 12:34:44.179093 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.179079    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ffec367-cdea-4176-adb9-4edf653641be" containerName="payload-processing"
Sep 09 12:34:44.179093 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.179090    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ffec367-cdea-4176-adb9-4edf653641be" containerName="payload-processing"
Sep 09 12:34:44.179169 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.179106    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4359a90f-0dfd-46cd-8bf8-2c8160df83c3" containerName="maas-api"
Sep 09 12:34:44.179169 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.179111    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4359a90f-0dfd-46cd-8bf8-2c8160df83c3" containerName="maas-api"
Sep 09 12:34:44.179169 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.179123    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff803f24-ae09-407f-b7c0-be2b6c92eed0" containerName="payload-pre-processing"
Sep 09 12:34:44.179169 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.179128    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff803f24-ae09-407f-b7c0-be2b6c92eed0" containerName="payload-pre-processing"
Sep 09 12:34:44.179334 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.179196    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff803f24-ae09-407f-b7c0-be2b6c92eed0" containerName="payload-pre-processing"
Sep 09 12:34:44.179334 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.179203    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4359a90f-0dfd-46cd-8bf8-2c8160df83c3" containerName="maas-api"
Sep 09 12:34:44.179334 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.179209    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ffec367-cdea-4176-adb9-4edf653641be" containerName="payload-processing"
Sep 09 12:34:44.184123 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.184093    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.186166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.186143    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-aea1bdeb-serving-cert\""
Sep 09 12:34:44.192319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.192296    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"]
Sep 09 12:34:44.208927 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.208900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-maas-api-tls\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.209062 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.208932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54lkj\" (UniqueName: \"kubernetes.io/projected/4cd98905-b8ff-4599-9152-f971a675ef7f-kube-api-access-54lkj\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.209062 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.208952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd98905-b8ff-4599-9152-f971a675ef7f-openshift-service-ca\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.209062 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.209013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-metrics-certs\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.310168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.310131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-maas-api-tls\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.310168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.310176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54lkj\" (UniqueName: \"kubernetes.io/projected/4cd98905-b8ff-4599-9152-f971a675ef7f-kube-api-access-54lkj\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.310437 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.310197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd98905-b8ff-4599-9152-f971a675ef7f-openshift-service-ca\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.310437 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:34:44.310322    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-serving-cert: secret "maas-api-e2e-ait-aea1bdeb-serving-cert" not found
Sep 09 12:34:44.310437 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.310344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-metrics-certs\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.310437 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:34:44.310406    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-maas-api-tls podName:4cd98905-b8ff-4599-9152-f971a675ef7f nodeName:}" failed. No retries permitted until 2026-09-09 12:34:44.810383826 +0000 UTC m=+905.153981753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-maas-api-tls") pod "maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5" (UID: "4cd98905-b8ff-4599-9152-f971a675ef7f") : secret "maas-api-e2e-ait-aea1bdeb-serving-cert" not found
Sep 09 12:34:44.310857 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.310838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd98905-b8ff-4599-9152-f971a675ef7f-openshift-service-ca\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.313010 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.312993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-metrics-certs\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.319021 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.319000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54lkj\" (UniqueName: \"kubernetes.io/projected/4cd98905-b8ff-4599-9152-f971a675ef7f-kube-api-access-54lkj\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.816178 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.816138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-maas-api-tls\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.818628 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.818608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-maas-api-tls\") pod \"maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:44.857155 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.857122    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"]
Sep 09 12:34:44.860863 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.860849    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:34:44.862730 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.862709    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-aea1bdeb\""
Sep 09 12:34:44.871982 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.871960    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"]
Sep 09 12:34:44.890497 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.890475    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"]
Sep 09 12:34:44.893938 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.893923    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:34:44.895805 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.895788    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-aea1bdeb-dockercfg-vnfsd\""
Sep 09 12:34:44.904851 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.904828    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"]
Sep 09 12:34:44.917452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.917418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3871d7-f16d-4241-9431-0a46f5cee4f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c\" (UID: \"fd3871d7-f16d-4241-9431-0a46f5cee4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:34:44.917565 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.917466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22chp\" (UniqueName: \"kubernetes.io/projected/fd3871d7-f16d-4241-9431-0a46f5cee4f0-kube-api-access-22chp\") pod \"payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c\" (UID: \"fd3871d7-f16d-4241-9431-0a46f5cee4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:34:44.917565 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.917539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8e2acf-c651-4418-94c8-1b025ff161a6-plugins-config-volume\") pod \"payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk\" (UID: \"7b8e2acf-c651-4418-94c8-1b025ff161a6\") " pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:34:44.917643 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:44.917569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcs24\" (UniqueName: \"kubernetes.io/projected/7b8e2acf-c651-4418-94c8-1b025ff161a6-kube-api-access-jcs24\") pod \"payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk\" (UID: \"7b8e2acf-c651-4418-94c8-1b025ff161a6\") " pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:34:45.019153 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.019121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3871d7-f16d-4241-9431-0a46f5cee4f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c\" (UID: \"fd3871d7-f16d-4241-9431-0a46f5cee4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:34:45.019343 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.019163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22chp\" (UniqueName: \"kubernetes.io/projected/fd3871d7-f16d-4241-9431-0a46f5cee4f0-kube-api-access-22chp\") pod \"payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c\" (UID: \"fd3871d7-f16d-4241-9431-0a46f5cee4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:34:45.019343 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.019205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8e2acf-c651-4418-94c8-1b025ff161a6-plugins-config-volume\") pod \"payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk\" (UID: \"7b8e2acf-c651-4418-94c8-1b025ff161a6\") " pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:34:45.019343 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.019247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcs24\" (UniqueName: \"kubernetes.io/projected/7b8e2acf-c651-4418-94c8-1b025ff161a6-kube-api-access-jcs24\") pod \"payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk\" (UID: \"7b8e2acf-c651-4418-94c8-1b025ff161a6\") " pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:34:45.019969 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.019947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3871d7-f16d-4241-9431-0a46f5cee4f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c\" (UID: \"fd3871d7-f16d-4241-9431-0a46f5cee4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:34:45.020012 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.019961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8e2acf-c651-4418-94c8-1b025ff161a6-plugins-config-volume\") pod \"payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk\" (UID: \"7b8e2acf-c651-4418-94c8-1b025ff161a6\") " pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:34:45.027789 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.027761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22chp\" (UniqueName: \"kubernetes.io/projected/fd3871d7-f16d-4241-9431-0a46f5cee4f0-kube-api-access-22chp\") pod \"payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c\" (UID: \"fd3871d7-f16d-4241-9431-0a46f5cee4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:34:45.027945 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.027924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcs24\" (UniqueName: \"kubernetes.io/projected/7b8e2acf-c651-4418-94c8-1b025ff161a6-kube-api-access-jcs24\") pod \"payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk\" (UID: \"7b8e2acf-c651-4418-94c8-1b025ff161a6\") " pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:34:45.098621 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.098533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:45.172341 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.172307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:34:45.203895 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.203866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:34:45.370531 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.370401    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"]
Sep 09 12:34:45.373369 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:34:45.373337    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd98905_b8ff_4599_9152_f971a675ef7f.slice/crio-1a9eee8434b6f3a442caf305ab3f730a45455104c42bb6133092d285dcb816bf WatchSource:0}: Error finding container 1a9eee8434b6f3a442caf305ab3f730a45455104c42bb6133092d285dcb816bf: Status 404 returned error can't find the container with id 1a9eee8434b6f3a442caf305ab3f730a45455104c42bb6133092d285dcb816bf
Sep 09 12:34:45.455534 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.455505    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"]
Sep 09 12:34:45.457475 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:34:45.457440    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3871d7_f16d_4241_9431_0a46f5cee4f0.slice/crio-5935510078ec620b6e1792d692dd243962c34559ffca096766dec14ede736451 WatchSource:0}: Error finding container 5935510078ec620b6e1792d692dd243962c34559ffca096766dec14ede736451: Status 404 returned error can't find the container with id 5935510078ec620b6e1792d692dd243962c34559ffca096766dec14ede736451
Sep 09 12:34:45.486736 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.486708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"]
Sep 09 12:34:45.499426 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:34:45.499398    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8e2acf_c651_4418_94c8_1b025ff161a6.slice/crio-13e69961838d5c24dd56c5f451c7f821736b7b0e3e85304bfddd1b5a2efb8959 WatchSource:0}: Error finding container 13e69961838d5c24dd56c5f451c7f821736b7b0e3e85304bfddd1b5a2efb8959: Status 404 returned error can't find the container with id 13e69961838d5c24dd56c5f451c7f821736b7b0e3e85304bfddd1b5a2efb8959
Sep 09 12:34:45.672482 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.672447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk" event={"ID":"7b8e2acf-c651-4418-94c8-1b025ff161a6","Type":"ContainerStarted","Data":"027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372"}
Sep 09 12:34:45.672682 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.672505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk" event={"ID":"7b8e2acf-c651-4418-94c8-1b025ff161a6","Type":"ContainerStarted","Data":"13e69961838d5c24dd56c5f451c7f821736b7b0e3e85304bfddd1b5a2efb8959"}
Sep 09 12:34:45.672682 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.672531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:34:45.674099 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.674069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c" event={"ID":"fd3871d7-f16d-4241-9431-0a46f5cee4f0","Type":"ContainerStarted","Data":"9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676"}
Sep 09 12:34:45.674251 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.674116    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:34:45.674251 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.674139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c" event={"ID":"fd3871d7-f16d-4241-9431-0a46f5cee4f0","Type":"ContainerStarted","Data":"5935510078ec620b6e1792d692dd243962c34559ffca096766dec14ede736451"}
Sep 09 12:34:45.675395 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.675374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5" event={"ID":"4cd98905-b8ff-4599-9152-f971a675ef7f","Type":"ContainerStarted","Data":"1a9eee8434b6f3a442caf305ab3f730a45455104c42bb6133092d285dcb816bf"}
Sep 09 12:34:45.692272 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.692204    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk" podStartSLOduration=1.692186296 podStartE2EDuration="1.692186296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:34:45.691085958 +0000 UTC m=+906.034683907" watchObservedRunningTime="2026-09-09 12:34:45.692186296 +0000 UTC m=+906.035784295"
Sep 09 12:34:45.711031 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:45.710978    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c" podStartSLOduration=1.710957901 podStartE2EDuration="1.710957901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:34:45.710558178 +0000 UTC m=+906.054156127" watchObservedRunningTime="2026-09-09 12:34:45.710957901 +0000 UTC m=+906.054555852"
Sep 09 12:34:46.683192 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:46.683153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5" event={"ID":"4cd98905-b8ff-4599-9152-f971a675ef7f","Type":"ContainerStarted","Data":"374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8"}
Sep 09 12:34:46.683699 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:46.683479    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:46.703043 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:46.702992    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5" podStartSLOduration=2.067858055 podStartE2EDuration="2.702974434s" totalImagesPullingTime="635.116379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:34:46.701207117 +0000 UTC m=+907.044805065" watchObservedRunningTime="2026-09-09 12:34:46.702974434 +0000 UTC m=+907.046572381"
Sep 09 12:34:49.227797 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:49.227766    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:34:52.694157 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:52.694124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:34:56.685753 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:56.685720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:34:56.686194 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:34:56.685954    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:35:04.932427 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:04.932389    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:35:14.956145 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:14.956109    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"]
Sep 09 12:35:14.956771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:14.956370    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5" podUID="4cd98905-b8ff-4599-9152-f971a675ef7f" containerName="maas-api" containerID="cri-o://374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8" gracePeriod=30
Sep 09 12:35:15.022142 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.017524    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"]
Sep 09 12:35:15.022142 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.017848    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk" podUID="7b8e2acf-c651-4418-94c8-1b025ff161a6" containerName="payload-processing" containerID="cri-o://027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372" gracePeriod=30
Sep 09 12:35:15.027116 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.027083    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"]
Sep 09 12:35:15.027423 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.027376    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c" podUID="fd3871d7-f16d-4241-9431-0a46f5cee4f0" containerName="payload-pre-processing" containerID="cri-o://9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676" gracePeriod=30
Sep 09 12:35:15.281829 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.281802    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:35:15.289376 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.289351    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:35:15.297571 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.297535    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:35:15.320547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.320518    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcs24\" (UniqueName: \"kubernetes.io/projected/7b8e2acf-c651-4418-94c8-1b025ff161a6-kube-api-access-jcs24\") pod \"7b8e2acf-c651-4418-94c8-1b025ff161a6\" (UID: \"7b8e2acf-c651-4418-94c8-1b025ff161a6\") "
Sep 09 12:35:15.320547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.320552    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-maas-api-tls\") pod \"4cd98905-b8ff-4599-9152-f971a675ef7f\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") "
Sep 09 12:35:15.320797 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.320602    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22chp\" (UniqueName: \"kubernetes.io/projected/fd3871d7-f16d-4241-9431-0a46f5cee4f0-kube-api-access-22chp\") pod \"fd3871d7-f16d-4241-9431-0a46f5cee4f0\" (UID: \"fd3871d7-f16d-4241-9431-0a46f5cee4f0\") "
Sep 09 12:35:15.320797 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.320654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3871d7-f16d-4241-9431-0a46f5cee4f0-plugins-config-volume\") pod \"fd3871d7-f16d-4241-9431-0a46f5cee4f0\" (UID: \"fd3871d7-f16d-4241-9431-0a46f5cee4f0\") "
Sep 09 12:35:15.322721 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.320792    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54lkj\" (UniqueName: \"kubernetes.io/projected/4cd98905-b8ff-4599-9152-f971a675ef7f-kube-api-access-54lkj\") pod \"4cd98905-b8ff-4599-9152-f971a675ef7f\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") "
Sep 09 12:35:15.322721 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.320840    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8e2acf-c651-4418-94c8-1b025ff161a6-plugins-config-volume\") pod \"7b8e2acf-c651-4418-94c8-1b025ff161a6\" (UID: \"7b8e2acf-c651-4418-94c8-1b025ff161a6\") "
Sep 09 12:35:15.322721 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.320868    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-metrics-certs\") pod \"4cd98905-b8ff-4599-9152-f971a675ef7f\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") "
Sep 09 12:35:15.322721 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.320998    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd98905-b8ff-4599-9152-f971a675ef7f-openshift-service-ca\") pod \"4cd98905-b8ff-4599-9152-f971a675ef7f\" (UID: \"4cd98905-b8ff-4599-9152-f971a675ef7f\") "
Sep 09 12:35:15.322721 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.321878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8e2acf-c651-4418-94c8-1b025ff161a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b8e2acf-c651-4418-94c8-1b025ff161a6" (UID: "7b8e2acf-c651-4418-94c8-1b025ff161a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:35:15.322721 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.321906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd98905-b8ff-4599-9152-f971a675ef7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4cd98905-b8ff-4599-9152-f971a675ef7f" (UID: "4cd98905-b8ff-4599-9152-f971a675ef7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:35:15.324289 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.323892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8e2acf-c651-4418-94c8-1b025ff161a6-kube-api-access-jcs24" (OuterVolumeSpecName: "kube-api-access-jcs24") pod "7b8e2acf-c651-4418-94c8-1b025ff161a6" (UID: "7b8e2acf-c651-4418-94c8-1b025ff161a6"). InnerVolumeSpecName "kube-api-access-jcs24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:35:15.324640 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.324589    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3871d7-f16d-4241-9431-0a46f5cee4f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd3871d7-f16d-4241-9431-0a46f5cee4f0" (UID: "fd3871d7-f16d-4241-9431-0a46f5cee4f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:35:15.324815 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.324794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4cd98905-b8ff-4599-9152-f971a675ef7f" (UID: "4cd98905-b8ff-4599-9152-f971a675ef7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:35:15.325281 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.325253    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4cd98905-b8ff-4599-9152-f971a675ef7f" (UID: "4cd98905-b8ff-4599-9152-f971a675ef7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:35:15.325398 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.325368    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3871d7-f16d-4241-9431-0a46f5cee4f0-kube-api-access-22chp" (OuterVolumeSpecName: "kube-api-access-22chp") pod "fd3871d7-f16d-4241-9431-0a46f5cee4f0" (UID: "fd3871d7-f16d-4241-9431-0a46f5cee4f0"). InnerVolumeSpecName "kube-api-access-22chp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:35:15.326064 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.326039    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd98905-b8ff-4599-9152-f971a675ef7f-kube-api-access-54lkj" (OuterVolumeSpecName: "kube-api-access-54lkj") pod "4cd98905-b8ff-4599-9152-f971a675ef7f" (UID: "4cd98905-b8ff-4599-9152-f971a675ef7f"). InnerVolumeSpecName "kube-api-access-54lkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:35:15.422660 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.422622    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd98905-b8ff-4599-9152-f971a675ef7f-openshift-service-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:15.422660 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.422656    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcs24\" (UniqueName: \"kubernetes.io/projected/7b8e2acf-c651-4418-94c8-1b025ff161a6-kube-api-access-jcs24\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:15.422660 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.422666    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-maas-api-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:15.422896 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.422675    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22chp\" (UniqueName: \"kubernetes.io/projected/fd3871d7-f16d-4241-9431-0a46f5cee4f0-kube-api-access-22chp\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:15.422896 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.422683    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3871d7-f16d-4241-9431-0a46f5cee4f0-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:15.422896 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.422691    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54lkj\" (UniqueName: \"kubernetes.io/projected/4cd98905-b8ff-4599-9152-f971a675ef7f-kube-api-access-54lkj\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:15.422896 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.422699    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8e2acf-c651-4418-94c8-1b025ff161a6-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:15.422896 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.422707    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd98905-b8ff-4599-9152-f971a675ef7f-metrics-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:15.838094 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.838061    2578 generic.go:356] "Generic (PLEG): container finished" podID="fd3871d7-f16d-4241-9431-0a46f5cee4f0" containerID="9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676" exitCode=0
Sep 09 12:35:15.838311 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.838104    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"
Sep 09 12:35:15.838311 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.838139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c" event={"ID":"fd3871d7-f16d-4241-9431-0a46f5cee4f0","Type":"ContainerDied","Data":"9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676"}
Sep 09 12:35:15.838311 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.838190    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c" event={"ID":"fd3871d7-f16d-4241-9431-0a46f5cee4f0","Type":"ContainerDied","Data":"5935510078ec620b6e1792d692dd243962c34559ffca096766dec14ede736451"}
Sep 09 12:35:15.838311 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.838241    2578 scope.go:117] "RemoveContainer" containerID="9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676"
Sep 09 12:35:15.839595 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.839576    2578 generic.go:356] "Generic (PLEG): container finished" podID="4cd98905-b8ff-4599-9152-f971a675ef7f" containerID="374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8" exitCode=0
Sep 09 12:35:15.839691 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.839676    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"
Sep 09 12:35:15.839748 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.839716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5" event={"ID":"4cd98905-b8ff-4599-9152-f971a675ef7f","Type":"ContainerDied","Data":"374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8"}
Sep 09 12:35:15.839800 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.839752    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5" event={"ID":"4cd98905-b8ff-4599-9152-f971a675ef7f","Type":"ContainerDied","Data":"1a9eee8434b6f3a442caf305ab3f730a45455104c42bb6133092d285dcb816bf"}
Sep 09 12:35:15.841119 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.841090    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b8e2acf-c651-4418-94c8-1b025ff161a6" containerID="027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372" exitCode=0
Sep 09 12:35:15.841269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.841117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk" event={"ID":"7b8e2acf-c651-4418-94c8-1b025ff161a6","Type":"ContainerDied","Data":"027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372"}
Sep 09 12:35:15.841269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.841146    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"
Sep 09 12:35:15.841269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.841206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk" event={"ID":"7b8e2acf-c651-4418-94c8-1b025ff161a6","Type":"ContainerDied","Data":"13e69961838d5c24dd56c5f451c7f821736b7b0e3e85304bfddd1b5a2efb8959"}
Sep 09 12:35:15.849968 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.849945    2578 scope.go:117] "RemoveContainer" containerID="9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676"
Sep 09 12:35:15.850302 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:35:15.850276    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676\": container with ID starting with 9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676 not found: ID does not exist" containerID="9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676"
Sep 09 12:35:15.850673 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.850315    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676"} err="failed to get container status \"9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676\": rpc error: code = NotFound desc = could not find container \"9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676\": container with ID starting with 9403c13840ecfdc649f8c267eb476db907e2e5971a054e5eb4099f49730c5676 not found: ID does not exist"
Sep 09 12:35:15.850673 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.850338    2578 scope.go:117] "RemoveContainer" containerID="374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8"
Sep 09 12:35:15.859869 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.859456    2578 scope.go:117] "RemoveContainer" containerID="374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8"
Sep 09 12:35:15.859987 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:35:15.859964    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8\": container with ID starting with 374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8 not found: ID does not exist" containerID="374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8"
Sep 09 12:35:15.860050 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.859998    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8"} err="failed to get container status \"374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8\": rpc error: code = NotFound desc = could not find container \"374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8\": container with ID starting with 374d4072689b16362cb8e89de7e6ac33d754aece742644ce0799d53e50d6a2a8 not found: ID does not exist"
Sep 09 12:35:15.860050 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.860020    2578 scope.go:117] "RemoveContainer" containerID="027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372"
Sep 09 12:35:15.861884 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.861866    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"]
Sep 09 12:35:15.869058 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.869035    2578 scope.go:117] "RemoveContainer" containerID="027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372"
Sep 09 12:35:15.869473 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:35:15.869452    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372\": container with ID starting with 027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372 not found: ID does not exist" containerID="027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372"
Sep 09 12:35:15.869565 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.869480    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372"} err="failed to get container status \"027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372\": rpc error: code = NotFound desc = could not find container \"027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372\": container with ID starting with 027d9ac60456c1daa90efdf975f56dc7255da9d8647df282bd379622697ae372 not found: ID does not exist"
Sep 09 12:35:15.870692 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.870671    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aea1bdeb-74c954f554-l545c"]
Sep 09 12:35:15.882178 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.882153    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"]
Sep 09 12:35:15.886259 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.886239    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aea1bdeb-6748f665cf-fqht5"]
Sep 09 12:35:15.896531 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.896510    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"]
Sep 09 12:35:15.902511 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:15.902493    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aea1bdeb-5477696677-cdfvk"]
Sep 09 12:35:16.186198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:16.186172    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cd98905-b8ff-4599-9152-f971a675ef7f" path="/var/lib/kubelet/pods/4cd98905-b8ff-4599-9152-f971a675ef7f/volumes"
Sep 09 12:35:16.186588 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:16.186577    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b8e2acf-c651-4418-94c8-1b025ff161a6" path="/var/lib/kubelet/pods/7b8e2acf-c651-4418-94c8-1b025ff161a6/volumes"
Sep 09 12:35:16.186895 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:16.186882    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd3871d7-f16d-4241-9431-0a46f5cee4f0" path="/var/lib/kubelet/pods/fd3871d7-f16d-4241-9431-0a46f5cee4f0/volumes"
Sep 09 12:35:20.359067 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.359033    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"]
Sep 09 12:35:20.361424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.359450    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b8e2acf-c651-4418-94c8-1b025ff161a6" containerName="payload-processing"
Sep 09 12:35:20.361424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.359462    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b8e2acf-c651-4418-94c8-1b025ff161a6" containerName="payload-processing"
Sep 09 12:35:20.361424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.359472    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3871d7-f16d-4241-9431-0a46f5cee4f0" containerName="payload-pre-processing"
Sep 09 12:35:20.361424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.359477    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3871d7-f16d-4241-9431-0a46f5cee4f0" containerName="payload-pre-processing"
Sep 09 12:35:20.361424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.359485    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cd98905-b8ff-4599-9152-f971a675ef7f" containerName="maas-api"
Sep 09 12:35:20.361424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.359491    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cd98905-b8ff-4599-9152-f971a675ef7f" containerName="maas-api"
Sep 09 12:35:20.361424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.359558    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b8e2acf-c651-4418-94c8-1b025ff161a6" containerName="payload-processing"
Sep 09 12:35:20.361424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.359565    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cd98905-b8ff-4599-9152-f971a675ef7f" containerName="maas-api"
Sep 09 12:35:20.361424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.359581    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd3871d7-f16d-4241-9431-0a46f5cee4f0" containerName="payload-pre-processing"
Sep 09 12:35:20.362531 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.362516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.365676 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.365655    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ace3fa44-serving-cert\""
Sep 09 12:35:20.376710 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.376685    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"]
Sep 09 12:35:20.459123 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.459088    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:35:20.469916 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.469887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtcg5\" (UniqueName: \"kubernetes.io/projected/15e18b8e-886d-4640-b968-1c8b01012115-kube-api-access-mtcg5\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.470069 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.469925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e18b8e-886d-4640-b968-1c8b01012115-openshift-service-ca\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.470069 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.470010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-maas-api-tls\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.470069 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.470060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-metrics-certs\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.571491 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.571460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-maas-api-tls\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.571708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.571535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-metrics-certs\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.571708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.571594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtcg5\" (UniqueName: \"kubernetes.io/projected/15e18b8e-886d-4640-b968-1c8b01012115-kube-api-access-mtcg5\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.571708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.571616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e18b8e-886d-4640-b968-1c8b01012115-openshift-service-ca\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.572361 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.572315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e18b8e-886d-4640-b968-1c8b01012115-openshift-service-ca\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.574115 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.574095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-maas-api-tls\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.574196 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.574138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-metrics-certs\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.579258 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.579204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtcg5\" (UniqueName: \"kubernetes.io/projected/15e18b8e-886d-4640-b968-1c8b01012115-kube-api-access-mtcg5\") pod \"maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.672773 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.672741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:20.948541 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:20.948515    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"]
Sep 09 12:35:20.951266 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:35:20.951191    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e18b8e_886d_4640_b968_1c8b01012115.slice/crio-8611e9c77be70c6d2b2bf2a5f06056eff317fb5c83e2fd6f48d08b6c04f2912a WatchSource:0}: Error finding container 8611e9c77be70c6d2b2bf2a5f06056eff317fb5c83e2fd6f48d08b6c04f2912a: Status 404 returned error can't find the container with id 8611e9c77be70c6d2b2bf2a5f06056eff317fb5c83e2fd6f48d08b6c04f2912a
Sep 09 12:35:21.087357 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.087325    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"]
Sep 09 12:35:21.091964 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.091942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:21.093981 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.093950    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ace3fa44\""
Sep 09 12:35:21.094116 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.094017    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ace3fa44-dockercfg-9nnvn\""
Sep 09 12:35:21.101681 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.101659    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"]
Sep 09 12:35:21.133611 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.133581    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"]
Sep 09 12:35:21.137087 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.137072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:21.147760 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.147733    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"]
Sep 09 12:35:21.177166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.177139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7f5p\" (UniqueName: \"kubernetes.io/projected/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-kube-api-access-z7f5p\") pod \"payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg\" (UID: \"4abd5a8b-d1fd-455e-acb8-b0327f393b9d\") " pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:21.177364 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.177183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66c2c\" (UniqueName: \"kubernetes.io/projected/929a1a7a-64e7-4f27-ae3e-82881c0b8423-kube-api-access-66c2c\") pod \"payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd\" (UID: \"929a1a7a-64e7-4f27-ae3e-82881c0b8423\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:21.177364 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.177263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929a1a7a-64e7-4f27-ae3e-82881c0b8423-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd\" (UID: \"929a1a7a-64e7-4f27-ae3e-82881c0b8423\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:21.177364 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.177288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-plugins-config-volume\") pod \"payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg\" (UID: \"4abd5a8b-d1fd-455e-acb8-b0327f393b9d\") " pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:21.278089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.277996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7f5p\" (UniqueName: \"kubernetes.io/projected/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-kube-api-access-z7f5p\") pod \"payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg\" (UID: \"4abd5a8b-d1fd-455e-acb8-b0327f393b9d\") " pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:21.278089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.278049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66c2c\" (UniqueName: \"kubernetes.io/projected/929a1a7a-64e7-4f27-ae3e-82881c0b8423-kube-api-access-66c2c\") pod \"payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd\" (UID: \"929a1a7a-64e7-4f27-ae3e-82881c0b8423\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:21.278089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.278092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929a1a7a-64e7-4f27-ae3e-82881c0b8423-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd\" (UID: \"929a1a7a-64e7-4f27-ae3e-82881c0b8423\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:21.278418 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.278118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-plugins-config-volume\") pod \"payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg\" (UID: \"4abd5a8b-d1fd-455e-acb8-b0327f393b9d\") " pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:21.278828 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.278806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929a1a7a-64e7-4f27-ae3e-82881c0b8423-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd\" (UID: \"929a1a7a-64e7-4f27-ae3e-82881c0b8423\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:21.278828 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.278817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-plugins-config-volume\") pod \"payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg\" (UID: \"4abd5a8b-d1fd-455e-acb8-b0327f393b9d\") " pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:21.286685 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.286658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7f5p\" (UniqueName: \"kubernetes.io/projected/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-kube-api-access-z7f5p\") pod \"payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg\" (UID: \"4abd5a8b-d1fd-455e-acb8-b0327f393b9d\") " pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:21.286799 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.286657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66c2c\" (UniqueName: \"kubernetes.io/projected/929a1a7a-64e7-4f27-ae3e-82881c0b8423-kube-api-access-66c2c\") pod \"payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd\" (UID: \"929a1a7a-64e7-4f27-ae3e-82881c0b8423\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:21.404332 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.404291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:21.447972 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.447929    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:21.688046 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.688015    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"]
Sep 09 12:35:21.689097 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:35:21.689072    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abd5a8b_d1fd_455e_acb8_b0327f393b9d.slice/crio-7e50afc4ae0311beb7918d7ddb9bef538e3bf547c22e039dcd7092db43d5ee8d WatchSource:0}: Error finding container 7e50afc4ae0311beb7918d7ddb9bef538e3bf547c22e039dcd7092db43d5ee8d: Status 404 returned error can't find the container with id 7e50afc4ae0311beb7918d7ddb9bef538e3bf547c22e039dcd7092db43d5ee8d
Sep 09 12:35:21.738025 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.738002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"]
Sep 09 12:35:21.740508 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:35:21.740484    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929a1a7a_64e7_4f27_ae3e_82881c0b8423.slice/crio-cb5fd77934786645c1cd38d9b33c38a23fec7187d79a1c14ac651ac4ea928cc9 WatchSource:0}: Error finding container cb5fd77934786645c1cd38d9b33c38a23fec7187d79a1c14ac651ac4ea928cc9: Status 404 returned error can't find the container with id cb5fd77934786645c1cd38d9b33c38a23fec7187d79a1c14ac651ac4ea928cc9
Sep 09 12:35:21.878293 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.878258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg" event={"ID":"4abd5a8b-d1fd-455e-acb8-b0327f393b9d","Type":"ContainerStarted","Data":"5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653"}
Sep 09 12:35:21.878492 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.878438    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:21.878492 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.878483    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg" event={"ID":"4abd5a8b-d1fd-455e-acb8-b0327f393b9d","Type":"ContainerStarted","Data":"7e50afc4ae0311beb7918d7ddb9bef538e3bf547c22e039dcd7092db43d5ee8d"}
Sep 09 12:35:21.880032 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.880004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2" event={"ID":"15e18b8e-886d-4640-b968-1c8b01012115","Type":"ContainerStarted","Data":"aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8"}
Sep 09 12:35:21.880168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.880057    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:21.880168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.880083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2" event={"ID":"15e18b8e-886d-4640-b968-1c8b01012115","Type":"ContainerStarted","Data":"8611e9c77be70c6d2b2bf2a5f06056eff317fb5c83e2fd6f48d08b6c04f2912a"}
Sep 09 12:35:21.881676 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.881651    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd" event={"ID":"929a1a7a-64e7-4f27-ae3e-82881c0b8423","Type":"ContainerStarted","Data":"214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a"}
Sep 09 12:35:21.881769 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.881700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd" event={"ID":"929a1a7a-64e7-4f27-ae3e-82881c0b8423","Type":"ContainerStarted","Data":"cb5fd77934786645c1cd38d9b33c38a23fec7187d79a1c14ac651ac4ea928cc9"}
Sep 09 12:35:21.881769 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.881727    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:21.894318 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.894254    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg" podStartSLOduration=0.89423602 podStartE2EDuration="894.23602ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:35:21.892908133 +0000 UTC m=+942.236506078" watchObservedRunningTime="2026-09-09 12:35:21.89423602 +0000 UTC m=+942.237833969"
Sep 09 12:35:21.908944 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.908894    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd" podStartSLOduration=0.908880863 podStartE2EDuration="908.880863ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:35:21.907255198 +0000 UTC m=+942.250853147" watchObservedRunningTime="2026-09-09 12:35:21.908880863 +0000 UTC m=+942.252478814"
Sep 09 12:35:21.922861 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:21.922808    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2" podStartSLOduration=1.4460000339999999 podStartE2EDuration="1.922791431s" totalImagesPullingTime="476.791397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:35:21.92190183 +0000 UTC m=+942.265499782" watchObservedRunningTime="2026-09-09 12:35:21.922791431 +0000 UTC m=+942.266389380"
Sep 09 12:35:27.893423 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:27.893394    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:32.889953 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:32.889923    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:32.890424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:32.890070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:38.927620 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:38.927576    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:35:51.042023 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.041979    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"]
Sep 09 12:35:51.042595 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.042205    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2" podUID="15e18b8e-886d-4640-b968-1c8b01012115" containerName="maas-api" containerID="cri-o://aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8" gracePeriod=30
Sep 09 12:35:51.086530 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.086496    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"]
Sep 09 12:35:51.086768 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.086739    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg" podUID="4abd5a8b-d1fd-455e-acb8-b0327f393b9d" containerName="payload-processing" containerID="cri-o://5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653" gracePeriod=30
Sep 09 12:35:51.097245 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.096691    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"]
Sep 09 12:35:51.097245 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.096921    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd" podUID="929a1a7a-64e7-4f27-ae3e-82881c0b8423" containerName="payload-pre-processing" containerID="cri-o://214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a" gracePeriod=30
Sep 09 12:35:51.372627 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.371884    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:51.381911 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.381884    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:51.407767 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.407676    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:51.476122 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.476086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7f5p\" (UniqueName: \"kubernetes.io/projected/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-kube-api-access-z7f5p\") pod \"4abd5a8b-d1fd-455e-acb8-b0327f393b9d\" (UID: \"4abd5a8b-d1fd-455e-acb8-b0327f393b9d\") "
Sep 09 12:35:51.476322 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.476154    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-plugins-config-volume\") pod \"4abd5a8b-d1fd-455e-acb8-b0327f393b9d\" (UID: \"4abd5a8b-d1fd-455e-acb8-b0327f393b9d\") "
Sep 09 12:35:51.476587 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.476560    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4abd5a8b-d1fd-455e-acb8-b0327f393b9d" (UID: "4abd5a8b-d1fd-455e-acb8-b0327f393b9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:35:51.478415 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.478395    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-kube-api-access-z7f5p" (OuterVolumeSpecName: "kube-api-access-z7f5p") pod "4abd5a8b-d1fd-455e-acb8-b0327f393b9d" (UID: "4abd5a8b-d1fd-455e-acb8-b0327f393b9d"). InnerVolumeSpecName "kube-api-access-z7f5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:35:51.576970 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.576877    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtcg5\" (UniqueName: \"kubernetes.io/projected/15e18b8e-886d-4640-b968-1c8b01012115-kube-api-access-mtcg5\") pod \"15e18b8e-886d-4640-b968-1c8b01012115\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") "
Sep 09 12:35:51.576970 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.576919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e18b8e-886d-4640-b968-1c8b01012115-openshift-service-ca\") pod \"15e18b8e-886d-4640-b968-1c8b01012115\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") "
Sep 09 12:35:51.577167 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.576980    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66c2c\" (UniqueName: \"kubernetes.io/projected/929a1a7a-64e7-4f27-ae3e-82881c0b8423-kube-api-access-66c2c\") pod \"929a1a7a-64e7-4f27-ae3e-82881c0b8423\" (UID: \"929a1a7a-64e7-4f27-ae3e-82881c0b8423\") "
Sep 09 12:35:51.577167 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.577004    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-metrics-certs\") pod \"15e18b8e-886d-4640-b968-1c8b01012115\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") "
Sep 09 12:35:51.577167 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.577054    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929a1a7a-64e7-4f27-ae3e-82881c0b8423-plugins-config-volume\") pod \"929a1a7a-64e7-4f27-ae3e-82881c0b8423\" (UID: \"929a1a7a-64e7-4f27-ae3e-82881c0b8423\") "
Sep 09 12:35:51.577167 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.577100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-maas-api-tls\") pod \"15e18b8e-886d-4640-b968-1c8b01012115\" (UID: \"15e18b8e-886d-4640-b968-1c8b01012115\") "
Sep 09 12:35:51.577436 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.577384    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7f5p\" (UniqueName: \"kubernetes.io/projected/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-kube-api-access-z7f5p\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:51.577436 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.577407    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4abd5a8b-d1fd-455e-acb8-b0327f393b9d-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:51.577436 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.577418    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e18b8e-886d-4640-b968-1c8b01012115-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15e18b8e-886d-4640-b968-1c8b01012115" (UID: "15e18b8e-886d-4640-b968-1c8b01012115"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:35:51.577584 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.577514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929a1a7a-64e7-4f27-ae3e-82881c0b8423-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "929a1a7a-64e7-4f27-ae3e-82881c0b8423" (UID: "929a1a7a-64e7-4f27-ae3e-82881c0b8423"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:35:51.579257 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.579209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e18b8e-886d-4640-b968-1c8b01012115-kube-api-access-mtcg5" (OuterVolumeSpecName: "kube-api-access-mtcg5") pod "15e18b8e-886d-4640-b968-1c8b01012115" (UID: "15e18b8e-886d-4640-b968-1c8b01012115"). InnerVolumeSpecName "kube-api-access-mtcg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:35:51.579387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.579365    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15e18b8e-886d-4640-b968-1c8b01012115" (UID: "15e18b8e-886d-4640-b968-1c8b01012115"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:35:51.579666 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.579650    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929a1a7a-64e7-4f27-ae3e-82881c0b8423-kube-api-access-66c2c" (OuterVolumeSpecName: "kube-api-access-66c2c") pod "929a1a7a-64e7-4f27-ae3e-82881c0b8423" (UID: "929a1a7a-64e7-4f27-ae3e-82881c0b8423"). InnerVolumeSpecName "kube-api-access-66c2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:35:51.579666 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.579657    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "15e18b8e-886d-4640-b968-1c8b01012115" (UID: "15e18b8e-886d-4640-b968-1c8b01012115"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:35:51.620004 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.619966    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"]
Sep 09 12:35:51.620419 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.620406    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e18b8e-886d-4640-b968-1c8b01012115" containerName="maas-api"
Sep 09 12:35:51.620473 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.620421    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e18b8e-886d-4640-b968-1c8b01012115" containerName="maas-api"
Sep 09 12:35:51.620473 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.620436    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929a1a7a-64e7-4f27-ae3e-82881c0b8423" containerName="payload-pre-processing"
Sep 09 12:35:51.620473 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.620441    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="929a1a7a-64e7-4f27-ae3e-82881c0b8423" containerName="payload-pre-processing"
Sep 09 12:35:51.620473 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.620457    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4abd5a8b-d1fd-455e-acb8-b0327f393b9d" containerName="payload-processing"
Sep 09 12:35:51.620473 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.620463    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4abd5a8b-d1fd-455e-acb8-b0327f393b9d" containerName="payload-processing"
Sep 09 12:35:51.620623 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.620520    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4abd5a8b-d1fd-455e-acb8-b0327f393b9d" containerName="payload-processing"
Sep 09 12:35:51.620623 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.620529    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="15e18b8e-886d-4640-b968-1c8b01012115" containerName="maas-api"
Sep 09 12:35:51.620623 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.620538    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="929a1a7a-64e7-4f27-ae3e-82881c0b8423" containerName="payload-pre-processing"
Sep 09 12:35:51.624052 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.624036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.626259 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.626235    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-e917b0\"/\"kube-root-ca.crt\""
Sep 09 12:35:51.626259 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.626256    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-e917b0\"/\"autores-model-e917b0-kserve-self-signed-certs\""
Sep 09 12:35:51.626439 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.626235    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-e917b0\"/\"default-dockercfg-bjfvx\""
Sep 09 12:35:51.626439 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.626234    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-e917b0\"/\"openshift-service-ca.crt\""
Sep 09 12:35:51.632793 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.632768    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"]
Sep 09 12:35:51.678495 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.678464    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-metrics-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:51.678495 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.678492    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929a1a7a-64e7-4f27-ae3e-82881c0b8423-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:51.678709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.678503    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15e18b8e-886d-4640-b968-1c8b01012115-maas-api-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:51.678709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.678513    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtcg5\" (UniqueName: \"kubernetes.io/projected/15e18b8e-886d-4640-b968-1c8b01012115-kube-api-access-mtcg5\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:51.678709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.678523    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e18b8e-886d-4640-b968-1c8b01012115-openshift-service-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:51.678709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.678531    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66c2c\" (UniqueName: \"kubernetes.io/projected/929a1a7a-64e7-4f27-ae3e-82881c0b8423-kube-api-access-66c2c\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:35:51.779330 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.779295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv4d2\" (UniqueName: \"kubernetes.io/projected/c209b853-6266-4a4b-a62c-e0613dc91a23-kube-api-access-zv4d2\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.779521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.779356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-tmp-dir\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.779521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.779389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-dshm\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.779521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.779418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-model-cache\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.779521 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.779487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c209b853-6266-4a4b-a62c-e0613dc91a23-tls-certs\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.779670 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.779525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-home\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.880598 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.880504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-tmp-dir\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.880598 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.880545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-dshm\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.880598 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.880572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-model-cache\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.880888 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.880628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c209b853-6266-4a4b-a62c-e0613dc91a23-tls-certs\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.880888 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.880679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-home\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.880888 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.880727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv4d2\" (UniqueName: \"kubernetes.io/projected/c209b853-6266-4a4b-a62c-e0613dc91a23-kube-api-access-zv4d2\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.881037 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.880968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-tmp-dir\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.881037 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.880978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-model-cache\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.881105 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.881086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-home\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.883095 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.883078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-dshm\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.883226 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.883197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c209b853-6266-4a4b-a62c-e0613dc91a23-tls-certs\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.890135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.890117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv4d2\" (UniqueName: \"kubernetes.io/projected/c209b853-6266-4a4b-a62c-e0613dc91a23-kube-api-access-zv4d2\") pod \"autores-model-e917b0-kserve-596bf89b4c-48cbp\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") " pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:51.934417 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:51.934391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:35:52.043657 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.043628    2578 generic.go:356] "Generic (PLEG): container finished" podID="4abd5a8b-d1fd-455e-acb8-b0327f393b9d" containerID="5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653" exitCode=0
Sep 09 12:35:52.044090 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.043681    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"
Sep 09 12:35:52.044090 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.043702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg" event={"ID":"4abd5a8b-d1fd-455e-acb8-b0327f393b9d","Type":"ContainerDied","Data":"5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653"}
Sep 09 12:35:52.044090 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.043756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg" event={"ID":"4abd5a8b-d1fd-455e-acb8-b0327f393b9d","Type":"ContainerDied","Data":"7e50afc4ae0311beb7918d7ddb9bef538e3bf547c22e039dcd7092db43d5ee8d"}
Sep 09 12:35:52.044090 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.043788    2578 scope.go:117] "RemoveContainer" containerID="5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653"
Sep 09 12:35:52.045181 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.045156    2578 generic.go:356] "Generic (PLEG): container finished" podID="15e18b8e-886d-4640-b968-1c8b01012115" containerID="aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8" exitCode=0
Sep 09 12:35:52.045297 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.045179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2" event={"ID":"15e18b8e-886d-4640-b968-1c8b01012115","Type":"ContainerDied","Data":"aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8"}
Sep 09 12:35:52.045297 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.045206    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"
Sep 09 12:35:52.045297 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.045249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2" event={"ID":"15e18b8e-886d-4640-b968-1c8b01012115","Type":"ContainerDied","Data":"8611e9c77be70c6d2b2bf2a5f06056eff317fb5c83e2fd6f48d08b6c04f2912a"}
Sep 09 12:35:52.046664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.046638    2578 generic.go:356] "Generic (PLEG): container finished" podID="929a1a7a-64e7-4f27-ae3e-82881c0b8423" containerID="214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a" exitCode=0
Sep 09 12:35:52.046792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.046677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd" event={"ID":"929a1a7a-64e7-4f27-ae3e-82881c0b8423","Type":"ContainerDied","Data":"214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a"}
Sep 09 12:35:52.046792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.046691    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"
Sep 09 12:35:52.046792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.046714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd" event={"ID":"929a1a7a-64e7-4f27-ae3e-82881c0b8423","Type":"ContainerDied","Data":"cb5fd77934786645c1cd38d9b33c38a23fec7187d79a1c14ac651ac4ea928cc9"}
Sep 09 12:35:52.053599 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.053585    2578 scope.go:117] "RemoveContainer" containerID="5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653"
Sep 09 12:35:52.053850 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:35:52.053827    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653\": container with ID starting with 5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653 not found: ID does not exist" containerID="5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653"
Sep 09 12:35:52.053906 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.053865    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653"} err="failed to get container status \"5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653\": rpc error: code = NotFound desc = could not find container \"5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653\": container with ID starting with 5f23e68d4978ab19ad085d6e0780ce740eca07c05089b0bcb4cadd3fee50f653 not found: ID does not exist"
Sep 09 12:35:52.053906 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.053892    2578 scope.go:117] "RemoveContainer" containerID="aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8"
Sep 09 12:35:52.072660 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.072643    2578 scope.go:117] "RemoveContainer" containerID="aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8"
Sep 09 12:35:52.072958 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:35:52.072940    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8\": container with ID starting with aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8 not found: ID does not exist" containerID="aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8"
Sep 09 12:35:52.073006 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.072964    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8"} err="failed to get container status \"aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8\": rpc error: code = NotFound desc = could not find container \"aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8\": container with ID starting with aa14e3f68dd1dbd4faf405c297543e9caecf74744dfc4283bc4c190a2b6212b8 not found: ID does not exist"
Sep 09 12:35:52.073006 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.072980    2578 scope.go:117] "RemoveContainer" containerID="214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a"
Sep 09 12:35:52.080986 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.080964    2578 scope.go:117] "RemoveContainer" containerID="214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a"
Sep 09 12:35:52.081409 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:35:52.081383    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a\": container with ID starting with 214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a not found: ID does not exist" containerID="214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a"
Sep 09 12:35:52.081493 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.081419    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a"} err="failed to get container status \"214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a\": rpc error: code = NotFound desc = could not find container \"214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a\": container with ID starting with 214925fd45ae7d91c8f68e6ccfe35e9308cfd1841b033b20cf11999e681f016a not found: ID does not exist"
Sep 09 12:35:52.088009 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.087987    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"]
Sep 09 12:35:52.100315 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.100285    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ace3fa44-796cdc8ff7-4hskg"]
Sep 09 12:35:52.109161 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.109138    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"]
Sep 09 12:35:52.113197 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.113175    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ace3fa44-58b9bf8844-sm7x2"]
Sep 09 12:35:52.122657 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.122635    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"]
Sep 09 12:35:52.131771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.131710    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ace3fa44-6cf676c66b-c8hgd"]
Sep 09 12:35:52.186270 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.186208    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15e18b8e-886d-4640-b968-1c8b01012115" path="/var/lib/kubelet/pods/15e18b8e-886d-4640-b968-1c8b01012115/volumes"
Sep 09 12:35:52.186676 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.186657    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4abd5a8b-d1fd-455e-acb8-b0327f393b9d" path="/var/lib/kubelet/pods/4abd5a8b-d1fd-455e-acb8-b0327f393b9d/volumes"
Sep 09 12:35:52.187151 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.187133    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="929a1a7a-64e7-4f27-ae3e-82881c0b8423" path="/var/lib/kubelet/pods/929a1a7a-64e7-4f27-ae3e-82881c0b8423/volumes"
Sep 09 12:35:52.210060 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:52.210034    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"]
Sep 09 12:35:52.210605 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:35:52.210582    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc209b853_6266_4a4b_a62c_e0613dc91a23.slice/crio-1c5a97107a70ca2dbe3012d36cd99e0a86592110c0ccbe47558fe3942c5a184b WatchSource:0}: Error finding container 1c5a97107a70ca2dbe3012d36cd99e0a86592110c0ccbe47558fe3942c5a184b: Status 404 returned error can't find the container with id 1c5a97107a70ca2dbe3012d36cd99e0a86592110c0ccbe47558fe3942c5a184b
Sep 09 12:35:53.054880 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:53.054839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp" event={"ID":"c209b853-6266-4a4b-a62c-e0613dc91a23","Type":"ContainerStarted","Data":"5c8c1248f99295205ce1e78503b55d76c76e5dd5dc29a0754186b1bb13e2c579"}
Sep 09 12:35:53.054880 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:53.054883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp" event={"ID":"c209b853-6266-4a4b-a62c-e0613dc91a23","Type":"ContainerStarted","Data":"1c5a97107a70ca2dbe3012d36cd99e0a86592110c0ccbe47558fe3942c5a184b"}
Sep 09 12:35:53.074226 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:53.074173    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp" podStartSLOduration=2.074158529 podStartE2EDuration="2.074158529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:35:53.072133036 +0000 UTC m=+973.415730984" watchObservedRunningTime="2026-09-09 12:35:53.074158529 +0000 UTC m=+973.417756478"
Sep 09 12:35:57.727827 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:35:57.727792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:36:01.935398 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:01.935319    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:36:01.935398 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:01.935371    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:36:01.947970 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:01.947942    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:36:02.112941 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:02.112914    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:36:13.525670 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.525634    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"]
Sep 09 12:36:13.529104 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.529089    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.532396 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.532374    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1e223f\"/\"openshift-service-ca.crt\""
Sep 09 12:36:13.532529 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.532428    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1e223f\"/\"kube-root-ca.crt\""
Sep 09 12:36:13.532603 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.532383    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1e223f\"/\"test-model-1e223f-kserve-self-signed-certs\""
Sep 09 12:36:13.532603 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.532558    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1e223f\"/\"default-dockercfg-zxm27\""
Sep 09 12:36:13.540564 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.540536    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"]
Sep 09 12:36:13.588144 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.588112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbjsx\" (UniqueName: \"kubernetes.io/projected/88193e18-eff3-4f59-a5dc-29928d04c336-kube-api-access-qbjsx\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.588294 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.588162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-home\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.588294 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.588182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-model-cache\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.588373 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.588299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88193e18-eff3-4f59-a5dc-29928d04c336-tls-certs\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.588373 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.588334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-tmp-dir\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.588436 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.588377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-dshm\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.689183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.689146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88193e18-eff3-4f59-a5dc-29928d04c336-tls-certs\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.689183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.689190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-tmp-dir\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.689416 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.689237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-dshm\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.689416 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.689282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjsx\" (UniqueName: \"kubernetes.io/projected/88193e18-eff3-4f59-a5dc-29928d04c336-kube-api-access-qbjsx\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.689416 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.689322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-home\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.689590 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.689491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-model-cache\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.689648 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.689626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-tmp-dir\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.689710 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.689693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-home\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.689833 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.689814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-model-cache\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.691730 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.691709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-dshm\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.691864 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.691849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88193e18-eff3-4f59-a5dc-29928d04c336-tls-certs\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.696748 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.696731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjsx\" (UniqueName: \"kubernetes.io/projected/88193e18-eff3-4f59-a5dc-29928d04c336-kube-api-access-qbjsx\") pod \"test-model-1e223f-kserve-78bdbb79b7-z2zx4\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") " pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:13.840136 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:13.840036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:14.110295 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:14.110277    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"]
Sep 09 12:36:14.112792 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:36:14.112764    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88193e18_eff3_4f59_a5dc_29928d04c336.slice/crio-cfa18f8364ab4e3fa8ae99f6764dc0be4a3a1db064bfdb9340c7935a115c7afc WatchSource:0}: Error finding container cfa18f8364ab4e3fa8ae99f6764dc0be4a3a1db064bfdb9340c7935a115c7afc: Status 404 returned error can't find the container with id cfa18f8364ab4e3fa8ae99f6764dc0be4a3a1db064bfdb9340c7935a115c7afc
Sep 09 12:36:14.161776 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:14.161751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4" event={"ID":"88193e18-eff3-4f59-a5dc-29928d04c336","Type":"ContainerStarted","Data":"cfa18f8364ab4e3fa8ae99f6764dc0be4a3a1db064bfdb9340c7935a115c7afc"}
Sep 09 12:36:15.167948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:15.167909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4" event={"ID":"88193e18-eff3-4f59-a5dc-29928d04c336","Type":"ContainerStarted","Data":"b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49"}
Sep 09 12:36:15.186167 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:15.186115    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4" podStartSLOduration=2.186099524 podStartE2EDuration="2.186099524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:36:15.184093213 +0000 UTC m=+995.527691164" watchObservedRunningTime="2026-09-09 12:36:15.186099524 +0000 UTC m=+995.529697471"
Sep 09 12:36:15.420414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:15.420332    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"]
Sep 09 12:36:17.021576 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:17.021533    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"]
Sep 09 12:36:19.930166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:19.930131    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:36:23.840276 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:23.840241    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:23.840765 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:23.840291    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:23.853008 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:23.852980    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:24.232809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:24.232777    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:36:30.441149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.441089    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv" podUID="8a270143-a64f-402b-bc8b-cfdd357cb616" containerName="main" containerID="cri-o://7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c" gracePeriod=45
Sep 09 12:36:30.685274 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.685235    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:36:30.769396 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.769321    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-dshm\") pod \"8a270143-a64f-402b-bc8b-cfdd357cb616\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") "
Sep 09 12:36:30.769396 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.769363    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7jvb\" (UniqueName: \"kubernetes.io/projected/8a270143-a64f-402b-bc8b-cfdd357cb616-kube-api-access-g7jvb\") pod \"8a270143-a64f-402b-bc8b-cfdd357cb616\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") "
Sep 09 12:36:30.769396 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.769384    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a270143-a64f-402b-bc8b-cfdd357cb616-tls-certs\") pod \"8a270143-a64f-402b-bc8b-cfdd357cb616\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") "
Sep 09 12:36:30.769674 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.769417    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-home\") pod \"8a270143-a64f-402b-bc8b-cfdd357cb616\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") "
Sep 09 12:36:30.769674 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.769475    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-tmp-dir\") pod \"8a270143-a64f-402b-bc8b-cfdd357cb616\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") "
Sep 09 12:36:30.769674 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.769521    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-model-cache\") pod \"8a270143-a64f-402b-bc8b-cfdd357cb616\" (UID: \"8a270143-a64f-402b-bc8b-cfdd357cb616\") "
Sep 09 12:36:30.769674 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.769636    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-home" (OuterVolumeSpecName: "home") pod "8a270143-a64f-402b-bc8b-cfdd357cb616" (UID: "8a270143-a64f-402b-bc8b-cfdd357cb616"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:30.769880 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.769798    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a270143-a64f-402b-bc8b-cfdd357cb616" (UID: "8a270143-a64f-402b-bc8b-cfdd357cb616"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:30.769935 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.769884    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a270143-a64f-402b-bc8b-cfdd357cb616" (UID: "8a270143-a64f-402b-bc8b-cfdd357cb616"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:30.770190 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.770071    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-tmp-dir\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:30.770268 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.770196    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-model-cache\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:30.770268 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.770208    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-home\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:30.771785 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.771763    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-dshm" (OuterVolumeSpecName: "dshm") pod "8a270143-a64f-402b-bc8b-cfdd357cb616" (UID: "8a270143-a64f-402b-bc8b-cfdd357cb616"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:30.771888 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.771792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a270143-a64f-402b-bc8b-cfdd357cb616-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a270143-a64f-402b-bc8b-cfdd357cb616" (UID: "8a270143-a64f-402b-bc8b-cfdd357cb616"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:36:30.771888 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.771823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a270143-a64f-402b-bc8b-cfdd357cb616-kube-api-access-g7jvb" (OuterVolumeSpecName: "kube-api-access-g7jvb") pod "8a270143-a64f-402b-bc8b-cfdd357cb616" (UID: "8a270143-a64f-402b-bc8b-cfdd357cb616"). InnerVolumeSpecName "kube-api-access-g7jvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:36:30.870743 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.870710    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a270143-a64f-402b-bc8b-cfdd357cb616-dshm\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:30.870743 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.870736    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7jvb\" (UniqueName: \"kubernetes.io/projected/8a270143-a64f-402b-bc8b-cfdd357cb616-kube-api-access-g7jvb\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:30.870743 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:30.870747    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a270143-a64f-402b-bc8b-cfdd357cb616-tls-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:31.265658 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:31.265623    2578 generic.go:356] "Generic (PLEG): container finished" podID="8a270143-a64f-402b-bc8b-cfdd357cb616" containerID="7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c" exitCode=0
Sep 09 12:36:31.265835 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:31.265669    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"
Sep 09 12:36:31.265835 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:31.265712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv" event={"ID":"8a270143-a64f-402b-bc8b-cfdd357cb616","Type":"ContainerDied","Data":"7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c"}
Sep 09 12:36:31.265835 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:31.265758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv" event={"ID":"8a270143-a64f-402b-bc8b-cfdd357cb616","Type":"ContainerDied","Data":"12f69640cfac542e388f249c1dd325730de67d2c427862be01ef4c1a909b4871"}
Sep 09 12:36:31.265835 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:31.265776    2578 scope.go:117] "RemoveContainer" containerID="7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c"
Sep 09 12:36:31.275681 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:31.275664    2578 scope.go:117] "RemoveContainer" containerID="7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c"
Sep 09 12:36:31.275951 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:36:31.275935    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c\": container with ID starting with 7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c not found: ID does not exist" containerID="7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c"
Sep 09 12:36:31.276008 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:31.275960    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c"} err="failed to get container status \"7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c\": rpc error: code = NotFound desc = could not find container \"7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c\": container with ID starting with 7efeb88abff971bfc833faf171c977334597d910e1735914fff8459ea38f616c not found: ID does not exist"
Sep 09 12:36:31.287725 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:31.287695    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"]
Sep 09 12:36:31.293159 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:31.293137    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-c356d768-kserve-84947675f9-xk5kv"]
Sep 09 12:36:32.046239 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.046185    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp" podUID="c209b853-6266-4a4b-a62c-e0613dc91a23" containerName="main" containerID="cri-o://5c8c1248f99295205ce1e78503b55d76c76e5dd5dc29a0754186b1bb13e2c579" gracePeriod=45
Sep 09 12:36:32.102286 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.102247    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp" podUID="c209b853-6266-4a4b-a62c-e0613dc91a23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 09 12:36:32.186804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.186774    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a270143-a64f-402b-bc8b-cfdd357cb616" path="/var/lib/kubelet/pods/8a270143-a64f-402b-bc8b-cfdd357cb616/volumes"
Sep 09 12:36:32.275867 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.274988    2578 generic.go:356] "Generic (PLEG): container finished" podID="c209b853-6266-4a4b-a62c-e0613dc91a23" containerID="5c8c1248f99295205ce1e78503b55d76c76e5dd5dc29a0754186b1bb13e2c579" exitCode=0
Sep 09 12:36:32.275867 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.275046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp" event={"ID":"c209b853-6266-4a4b-a62c-e0613dc91a23","Type":"ContainerDied","Data":"5c8c1248f99295205ce1e78503b55d76c76e5dd5dc29a0754186b1bb13e2c579"}
Sep 09 12:36:32.301975 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.301915    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:36:32.383781 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.383754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv4d2\" (UniqueName: \"kubernetes.io/projected/c209b853-6266-4a4b-a62c-e0613dc91a23-kube-api-access-zv4d2\") pod \"c209b853-6266-4a4b-a62c-e0613dc91a23\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") "
Sep 09 12:36:32.383945 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.383826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-home\") pod \"c209b853-6266-4a4b-a62c-e0613dc91a23\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") "
Sep 09 12:36:32.383945 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.383854    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-dshm\") pod \"c209b853-6266-4a4b-a62c-e0613dc91a23\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") "
Sep 09 12:36:32.383945 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.383885    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-model-cache\") pod \"c209b853-6266-4a4b-a62c-e0613dc91a23\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") "
Sep 09 12:36:32.383945 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.383922    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c209b853-6266-4a4b-a62c-e0613dc91a23-tls-certs\") pod \"c209b853-6266-4a4b-a62c-e0613dc91a23\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") "
Sep 09 12:36:32.384170 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.383967    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-tmp-dir\") pod \"c209b853-6266-4a4b-a62c-e0613dc91a23\" (UID: \"c209b853-6266-4a4b-a62c-e0613dc91a23\") "
Sep 09 12:36:32.384170 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.384063    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-home" (OuterVolumeSpecName: "home") pod "c209b853-6266-4a4b-a62c-e0613dc91a23" (UID: "c209b853-6266-4a4b-a62c-e0613dc91a23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:32.384170 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.384153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-model-cache" (OuterVolumeSpecName: "model-cache") pod "c209b853-6266-4a4b-a62c-e0613dc91a23" (UID: "c209b853-6266-4a4b-a62c-e0613dc91a23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:32.384350 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.384268    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c209b853-6266-4a4b-a62c-e0613dc91a23" (UID: "c209b853-6266-4a4b-a62c-e0613dc91a23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:32.384413 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.384371    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-home\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:32.384413 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.384383    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-model-cache\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:32.384413 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.384392    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-tmp-dir\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:32.386255 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.386232    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c209b853-6266-4a4b-a62c-e0613dc91a23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c209b853-6266-4a4b-a62c-e0613dc91a23" (UID: "c209b853-6266-4a4b-a62c-e0613dc91a23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:36:32.386487 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.386463    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-dshm" (OuterVolumeSpecName: "dshm") pod "c209b853-6266-4a4b-a62c-e0613dc91a23" (UID: "c209b853-6266-4a4b-a62c-e0613dc91a23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:32.386487 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.386478    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c209b853-6266-4a4b-a62c-e0613dc91a23-kube-api-access-zv4d2" (OuterVolumeSpecName: "kube-api-access-zv4d2") pod "c209b853-6266-4a4b-a62c-e0613dc91a23" (UID: "c209b853-6266-4a4b-a62c-e0613dc91a23"). InnerVolumeSpecName "kube-api-access-zv4d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:36:32.485804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.485768    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv4d2\" (UniqueName: \"kubernetes.io/projected/c209b853-6266-4a4b-a62c-e0613dc91a23-kube-api-access-zv4d2\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:32.485804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.485798    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c209b853-6266-4a4b-a62c-e0613dc91a23-dshm\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:32.485804 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:32.485807    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c209b853-6266-4a4b-a62c-e0613dc91a23-tls-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:36:33.282371 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:33.282342    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"
Sep 09 12:36:33.282766 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:33.282384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp" event={"ID":"c209b853-6266-4a4b-a62c-e0613dc91a23","Type":"ContainerDied","Data":"1c5a97107a70ca2dbe3012d36cd99e0a86592110c0ccbe47558fe3942c5a184b"}
Sep 09 12:36:33.282766 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:33.282433    2578 scope.go:117] "RemoveContainer" containerID="5c8c1248f99295205ce1e78503b55d76c76e5dd5dc29a0754186b1bb13e2c579"
Sep 09 12:36:33.302254 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:33.302230    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"]
Sep 09 12:36:33.309973 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:33.309954    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-e917b0/autores-model-e917b0-kserve-596bf89b4c-48cbp"]
Sep 09 12:36:34.187607 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:34.187563    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c209b853-6266-4a4b-a62c-e0613dc91a23" path="/var/lib/kubelet/pods/c209b853-6266-4a4b-a62c-e0613dc91a23/volumes"
Sep 09 12:36:35.535090 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:35.535050    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:36:40.358278 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.358232    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"]
Sep 09 12:36:40.358814 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.358794    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c209b853-6266-4a4b-a62c-e0613dc91a23" containerName="main"
Sep 09 12:36:40.358899 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.358817    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c209b853-6266-4a4b-a62c-e0613dc91a23" containerName="main"
Sep 09 12:36:40.358899 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.358838    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a270143-a64f-402b-bc8b-cfdd357cb616" containerName="main"
Sep 09 12:36:40.358899 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.358846    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a270143-a64f-402b-bc8b-cfdd357cb616" containerName="main"
Sep 09 12:36:40.359052 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.358950    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c209b853-6266-4a4b-a62c-e0613dc91a23" containerName="main"
Sep 09 12:36:40.359052 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.358962    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a270143-a64f-402b-bc8b-cfdd357cb616" containerName="main"
Sep 09 12:36:40.363269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.363250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:40.365224 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.365198    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-0f93fa-serving-cert\""
Sep 09 12:36:40.375184 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.375154    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"]
Sep 09 12:36:40.463989 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.463957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07cbcacf-e0ef-4629-908a-984ba481f064-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:40.464149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.464011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhlz\" (UniqueName: \"kubernetes.io/projected/07cbcacf-e0ef-4629-908a-984ba481f064-kube-api-access-jhhlz\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:40.464149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.464054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-metrics-certs\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:40.464149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.464093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:40.565253 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.565160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhlz\" (UniqueName: \"kubernetes.io/projected/07cbcacf-e0ef-4629-908a-984ba481f064-kube-api-access-jhhlz\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:40.565402 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.565246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-metrics-certs\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:40.565402 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.565319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:40.565402 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.565380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07cbcacf-e0ef-4629-908a-984ba481f064-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:40.565526 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:36:40.565454    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-serving-cert: secret "maas-api-e2e-explicit-0f93fa-serving-cert" not found
Sep 09 12:36:40.565563 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:36:40.565533    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls podName:07cbcacf-e0ef-4629-908a-984ba481f064 nodeName:}" failed. No retries permitted until 2026-09-09 12:36:41.065511044 +0000 UTC m=+1021.409108983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls") pod "maas-api-e2e-explicit-0f93fa-664b77648f-xpp74" (UID: "07cbcacf-e0ef-4629-908a-984ba481f064") : secret "maas-api-e2e-explicit-0f93fa-serving-cert" not found
Sep 09 12:36:40.566003 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.565987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07cbcacf-e0ef-4629-908a-984ba481f064-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:40.567915 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.567895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-metrics-certs\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:40.576522 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:40.576497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhlz\" (UniqueName: \"kubernetes.io/projected/07cbcacf-e0ef-4629-908a-984ba481f064-kube-api-access-jhhlz\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:41.069950 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.069912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:41.070140 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:36:41.070063    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-serving-cert: secret "maas-api-e2e-explicit-0f93fa-serving-cert" not found
Sep 09 12:36:41.070190 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:36:41.070143    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls podName:07cbcacf-e0ef-4629-908a-984ba481f064 nodeName:}" failed. No retries permitted until 2026-09-09 12:36:42.070128093 +0000 UTC m=+1022.413726018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls") pod "maas-api-e2e-explicit-0f93fa-664b77648f-xpp74" (UID: "07cbcacf-e0ef-4629-908a-984ba481f064") : secret "maas-api-e2e-explicit-0f93fa-serving-cert" not found
Sep 09 12:36:41.094529 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.094492    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"]
Sep 09 12:36:41.097880 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.097860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:36:41.099891 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.099870    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-0f93fa\""
Sep 09 12:36:41.107288 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.107268    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"]
Sep 09 12:36:41.122203 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.122175    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"]
Sep 09 12:36:41.125407 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.125390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:36:41.127339 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.127319    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-0f93fa-dockercfg-ghzjw\""
Sep 09 12:36:41.135766 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.135747    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"]
Sep 09 12:36:41.170798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.170766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmcqh\" (UniqueName: \"kubernetes.io/projected/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-kube-api-access-cmcqh\") pod \"payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f\" (UID: \"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:36:41.170964 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.170814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21841879-125f-4030-ba71-8accec6be937-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x\" (UID: \"21841879-125f-4030-ba71-8accec6be937\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:36:41.170964 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.170891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f\" (UID: \"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:36:41.171056 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.170963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbrn\" (UniqueName: \"kubernetes.io/projected/21841879-125f-4030-ba71-8accec6be937-kube-api-access-xnbrn\") pod \"payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x\" (UID: \"21841879-125f-4030-ba71-8accec6be937\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:36:41.272494 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.272458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmcqh\" (UniqueName: \"kubernetes.io/projected/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-kube-api-access-cmcqh\") pod \"payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f\" (UID: \"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:36:41.272685 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.272517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21841879-125f-4030-ba71-8accec6be937-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x\" (UID: \"21841879-125f-4030-ba71-8accec6be937\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:36:41.272685 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.272581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f\" (UID: \"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:36:41.272685 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.272650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbrn\" (UniqueName: \"kubernetes.io/projected/21841879-125f-4030-ba71-8accec6be937-kube-api-access-xnbrn\") pod \"payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x\" (UID: \"21841879-125f-4030-ba71-8accec6be937\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:36:41.273272 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.273246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21841879-125f-4030-ba71-8accec6be937-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x\" (UID: \"21841879-125f-4030-ba71-8accec6be937\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:36:41.273357 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.273274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f\" (UID: \"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:36:41.280731 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.280701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbrn\" (UniqueName: \"kubernetes.io/projected/21841879-125f-4030-ba71-8accec6be937-kube-api-access-xnbrn\") pod \"payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x\" (UID: \"21841879-125f-4030-ba71-8accec6be937\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:36:41.280818 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.280744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmcqh\" (UniqueName: \"kubernetes.io/projected/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-kube-api-access-cmcqh\") pod \"payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f\" (UID: \"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:36:41.410025 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.409998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:36:41.436180 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.436154    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:36:41.685826 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.685801    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"]
Sep 09 12:36:41.687783 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:36:41.687758    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21841879_125f_4030_ba71_8accec6be937.slice/crio-ccae6a4f7c1d7ade0e273beb0b498df282a8ef41e988e48d28a575181b352f22 WatchSource:0}: Error finding container ccae6a4f7c1d7ade0e273beb0b498df282a8ef41e988e48d28a575181b352f22: Status 404 returned error can't find the container with id ccae6a4f7c1d7ade0e273beb0b498df282a8ef41e988e48d28a575181b352f22
Sep 09 12:36:41.718406 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:41.718338    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"]
Sep 09 12:36:41.726578 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:36:41.726554    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8346db_d0ca_4c1f_88cb_ceb1c9fa5afb.slice/crio-a00ce4a679bd743502a2b465cd39243a89652e8eeff610fa1600b15d82753116 WatchSource:0}: Error finding container a00ce4a679bd743502a2b465cd39243a89652e8eeff610fa1600b15d82753116: Status 404 returned error can't find the container with id a00ce4a679bd743502a2b465cd39243a89652e8eeff610fa1600b15d82753116
Sep 09 12:36:42.080499 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.080417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:42.083080 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.083059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls\") pod \"maas-api-e2e-explicit-0f93fa-664b77648f-xpp74\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:42.176932 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.176882    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:42.333030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.332945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f" event={"ID":"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb","Type":"ContainerStarted","Data":"b5e42ab3e3d0fc48dc3f360fa83620f9aba8cd0ebd5b9bf4cad37228202250a1"}
Sep 09 12:36:42.333030 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.332994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f" event={"ID":"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb","Type":"ContainerStarted","Data":"a00ce4a679bd743502a2b465cd39243a89652e8eeff610fa1600b15d82753116"}
Sep 09 12:36:42.333277 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.333062    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:36:42.334746 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.334723    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x" event={"ID":"21841879-125f-4030-ba71-8accec6be937","Type":"ContainerStarted","Data":"aea731c31aa3f401daa7a69218be3ecd691913aacd330801dfc66884c2e844cd"}
Sep 09 12:36:42.334870 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.334768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:36:42.334870 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.334784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x" event={"ID":"21841879-125f-4030-ba71-8accec6be937","Type":"ContainerStarted","Data":"ccae6a4f7c1d7ade0e273beb0b498df282a8ef41e988e48d28a575181b352f22"}
Sep 09 12:36:42.351002 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.350958    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f" podStartSLOduration=1.350943768 podStartE2EDuration="1.350943768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:36:42.349351171 +0000 UTC m=+1022.692949131" watchObservedRunningTime="2026-09-09 12:36:42.350943768 +0000 UTC m=+1022.694541716"
Sep 09 12:36:42.365001 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.364957    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x" podStartSLOduration=1.364937765 podStartE2EDuration="1.364937765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:36:42.364145964 +0000 UTC m=+1022.707743916" watchObservedRunningTime="2026-09-09 12:36:42.364937765 +0000 UTC m=+1022.708535703"
Sep 09 12:36:42.450838 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.450814    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"]
Sep 09 12:36:42.453097 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:36:42.453076    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07cbcacf_e0ef_4629_908a_984ba481f064.slice/crio-2886417acdbf14ecfe455c3e8ea90142a5edb844cab90690c6cb68cb75646895 WatchSource:0}: Error finding container 2886417acdbf14ecfe455c3e8ea90142a5edb844cab90690c6cb68cb75646895: Status 404 returned error can't find the container with id 2886417acdbf14ecfe455c3e8ea90142a5edb844cab90690c6cb68cb75646895
Sep 09 12:36:42.454650 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:42.454628    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:36:43.342741 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:43.342614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74" event={"ID":"07cbcacf-e0ef-4629-908a-984ba481f064","Type":"ContainerStarted","Data":"29ae9c898a19b077e3a0e4664e8b4067b4d746174a78f967c0b04a52e125c2e9"}
Sep 09 12:36:43.342741 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:43.342676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74" event={"ID":"07cbcacf-e0ef-4629-908a-984ba481f064","Type":"ContainerStarted","Data":"2886417acdbf14ecfe455c3e8ea90142a5edb844cab90690c6cb68cb75646895"}
Sep 09 12:36:43.343026 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:43.343002    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:43.358906 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:43.358862    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74" podStartSLOduration=2.860233125 podStartE2EDuration="3.358847962s" totalImagesPullingTime="498.614837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:36:43.358188194 +0000 UTC m=+1023.701786146" watchObservedRunningTime="2026-09-09 12:36:43.358847962 +0000 UTC m=+1023.702445911"
Sep 09 12:36:49.353048 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:49.353017    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:36:53.345025 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:53.344985    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:36:53.345471 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:53.345050    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:36:53.427182 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:36:53.427144    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:37:07.927055 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:37:07.927018    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:37:27.733042 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:37:27.733012    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:37:52.859725 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:37:52.859637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:37:54.341247 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:37:54.341192    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"]
Sep 09 12:38:01.461551 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.461520    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"]
Sep 09 12:38:01.461999 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.461747    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74" podUID="07cbcacf-e0ef-4629-908a-984ba481f064" containerName="maas-api" containerID="cri-o://29ae9c898a19b077e3a0e4664e8b4067b4d746174a78f967c0b04a52e125c2e9" gracePeriod=30
Sep 09 12:38:01.516711 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.516679    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"]
Sep 09 12:38:01.516964 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.516923    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f" podUID="dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb" containerName="payload-processing" containerID="cri-o://b5e42ab3e3d0fc48dc3f360fa83620f9aba8cd0ebd5b9bf4cad37228202250a1" gracePeriod=30
Sep 09 12:38:01.523675 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.523646    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"]
Sep 09 12:38:01.524080 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.524046    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x" podUID="21841879-125f-4030-ba71-8accec6be937" containerName="payload-pre-processing" containerID="cri-o://aea731c31aa3f401daa7a69218be3ecd691913aacd330801dfc66884c2e844cd" gracePeriod=30
Sep 09 12:38:01.754807 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.754768    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb" containerID="b5e42ab3e3d0fc48dc3f360fa83620f9aba8cd0ebd5b9bf4cad37228202250a1" exitCode=0
Sep 09 12:38:01.755018 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.754861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f" event={"ID":"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb","Type":"ContainerDied","Data":"b5e42ab3e3d0fc48dc3f360fa83620f9aba8cd0ebd5b9bf4cad37228202250a1"}
Sep 09 12:38:01.757033 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.756832    2578 generic.go:356] "Generic (PLEG): container finished" podID="21841879-125f-4030-ba71-8accec6be937" containerID="aea731c31aa3f401daa7a69218be3ecd691913aacd330801dfc66884c2e844cd" exitCode=0
Sep 09 12:38:01.757033 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.756966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x" event={"ID":"21841879-125f-4030-ba71-8accec6be937","Type":"ContainerDied","Data":"aea731c31aa3f401daa7a69218be3ecd691913aacd330801dfc66884c2e844cd"}
Sep 09 12:38:01.759080 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.758996    2578 generic.go:356] "Generic (PLEG): container finished" podID="07cbcacf-e0ef-4629-908a-984ba481f064" containerID="29ae9c898a19b077e3a0e4664e8b4067b4d746174a78f967c0b04a52e125c2e9" exitCode=0
Sep 09 12:38:01.759236 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.759085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74" event={"ID":"07cbcacf-e0ef-4629-908a-984ba481f064","Type":"ContainerDied","Data":"29ae9c898a19b077e3a0e4664e8b4067b4d746174a78f967c0b04a52e125c2e9"}
Sep 09 12:38:01.788798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.788768    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:38:01.800950 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.800921    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:38:01.811824 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.811794    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:38:01.854046 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.854018    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmcqh\" (UniqueName: \"kubernetes.io/projected/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-kube-api-access-cmcqh\") pod \"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb\" (UID: \"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb\") "
Sep 09 12:38:01.854204 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.854098    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-plugins-config-volume\") pod \"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb\" (UID: \"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb\") "
Sep 09 12:38:01.854204 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.854181    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls\") pod \"07cbcacf-e0ef-4629-908a-984ba481f064\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") "
Sep 09 12:38:01.854340 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.854265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhhlz\" (UniqueName: \"kubernetes.io/projected/07cbcacf-e0ef-4629-908a-984ba481f064-kube-api-access-jhhlz\") pod \"07cbcacf-e0ef-4629-908a-984ba481f064\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") "
Sep 09 12:38:01.854394 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.854345    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-metrics-certs\") pod \"07cbcacf-e0ef-4629-908a-984ba481f064\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") "
Sep 09 12:38:01.854394 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.854371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07cbcacf-e0ef-4629-908a-984ba481f064-openshift-service-ca\") pod \"07cbcacf-e0ef-4629-908a-984ba481f064\" (UID: \"07cbcacf-e0ef-4629-908a-984ba481f064\") "
Sep 09 12:38:01.854530 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.854496    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb" (UID: "dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:38:01.854809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.854786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07cbcacf-e0ef-4629-908a-984ba481f064-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07cbcacf-e0ef-4629-908a-984ba481f064" (UID: "07cbcacf-e0ef-4629-908a-984ba481f064"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:38:01.854923 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.854896    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:01.854923 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.854914    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07cbcacf-e0ef-4629-908a-984ba481f064-openshift-service-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:01.856509 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.856482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07cbcacf-e0ef-4629-908a-984ba481f064" (UID: "07cbcacf-e0ef-4629-908a-984ba481f064"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:38:01.856669 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.856641    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07cbcacf-e0ef-4629-908a-984ba481f064" (UID: "07cbcacf-e0ef-4629-908a-984ba481f064"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:38:01.856750 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.856689    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07cbcacf-e0ef-4629-908a-984ba481f064-kube-api-access-jhhlz" (OuterVolumeSpecName: "kube-api-access-jhhlz") pod "07cbcacf-e0ef-4629-908a-984ba481f064" (UID: "07cbcacf-e0ef-4629-908a-984ba481f064"). InnerVolumeSpecName "kube-api-access-jhhlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:01.856842 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.856822    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-kube-api-access-cmcqh" (OuterVolumeSpecName: "kube-api-access-cmcqh") pod "dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb" (UID: "dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb"). InnerVolumeSpecName "kube-api-access-cmcqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:01.955453 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.955416    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21841879-125f-4030-ba71-8accec6be937-plugins-config-volume\") pod \"21841879-125f-4030-ba71-8accec6be937\" (UID: \"21841879-125f-4030-ba71-8accec6be937\") "
Sep 09 12:38:01.955453 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.955458    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnbrn\" (UniqueName: \"kubernetes.io/projected/21841879-125f-4030-ba71-8accec6be937-kube-api-access-xnbrn\") pod \"21841879-125f-4030-ba71-8accec6be937\" (UID: \"21841879-125f-4030-ba71-8accec6be937\") "
Sep 09 12:38:01.955676 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.955661    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-metrics-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:01.955715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.955675    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmcqh\" (UniqueName: \"kubernetes.io/projected/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb-kube-api-access-cmcqh\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:01.955715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.955685    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07cbcacf-e0ef-4629-908a-984ba481f064-maas-api-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:01.955715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.955694    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhhlz\" (UniqueName: \"kubernetes.io/projected/07cbcacf-e0ef-4629-908a-984ba481f064-kube-api-access-jhhlz\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:01.955810 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.955760    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21841879-125f-4030-ba71-8accec6be937-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21841879-125f-4030-ba71-8accec6be937" (UID: "21841879-125f-4030-ba71-8accec6be937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:38:01.957622 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:01.957590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21841879-125f-4030-ba71-8accec6be937-kube-api-access-xnbrn" (OuterVolumeSpecName: "kube-api-access-xnbrn") pod "21841879-125f-4030-ba71-8accec6be937" (UID: "21841879-125f-4030-ba71-8accec6be937"). InnerVolumeSpecName "kube-api-access-xnbrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:02.056732 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.056651    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21841879-125f-4030-ba71-8accec6be937-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:02.056732 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.056679    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbrn\" (UniqueName: \"kubernetes.io/projected/21841879-125f-4030-ba71-8accec6be937-kube-api-access-xnbrn\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:02.766083 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.765997    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"
Sep 09 12:38:02.766083 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.765988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f" event={"ID":"dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb","Type":"ContainerDied","Data":"a00ce4a679bd743502a2b465cd39243a89652e8eeff610fa1600b15d82753116"}
Sep 09 12:38:02.766637 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.766154    2578 scope.go:117] "RemoveContainer" containerID="b5e42ab3e3d0fc48dc3f360fa83620f9aba8cd0ebd5b9bf4cad37228202250a1"
Sep 09 12:38:02.767798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.767597    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"
Sep 09 12:38:02.767798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.767668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x" event={"ID":"21841879-125f-4030-ba71-8accec6be937","Type":"ContainerDied","Data":"ccae6a4f7c1d7ade0e273beb0b498df282a8ef41e988e48d28a575181b352f22"}
Sep 09 12:38:02.769412 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.769384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74" event={"ID":"07cbcacf-e0ef-4629-908a-984ba481f064","Type":"ContainerDied","Data":"2886417acdbf14ecfe455c3e8ea90142a5edb844cab90690c6cb68cb75646895"}
Sep 09 12:38:02.769412 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.769404    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"
Sep 09 12:38:02.776176 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.776157    2578 scope.go:117] "RemoveContainer" containerID="aea731c31aa3f401daa7a69218be3ecd691913aacd330801dfc66884c2e844cd"
Sep 09 12:38:02.787323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.787306    2578 scope.go:117] "RemoveContainer" containerID="29ae9c898a19b077e3a0e4664e8b4067b4d746174a78f967c0b04a52e125c2e9"
Sep 09 12:38:02.788883 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.788860    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"]
Sep 09 12:38:02.793075 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.793053    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f93fa-664b77648f-xpp74"]
Sep 09 12:38:02.802439 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.802416    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"]
Sep 09 12:38:02.806079 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.806058    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f93fa-647666cd78-wn89x"]
Sep 09 12:38:02.816106 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.816085    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"]
Sep 09 12:38:02.820280 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:02.820259    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f93fa-7f77b5c849-c9z7f"]
Sep 09 12:38:04.186032 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:04.186000    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07cbcacf-e0ef-4629-908a-984ba481f064" path="/var/lib/kubelet/pods/07cbcacf-e0ef-4629-908a-984ba481f064/volumes"
Sep 09 12:38:04.186446 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:04.186374    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21841879-125f-4030-ba71-8accec6be937" path="/var/lib/kubelet/pods/21841879-125f-4030-ba71-8accec6be937/volumes"
Sep 09 12:38:04.186687 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:04.186675    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb" path="/var/lib/kubelet/pods/dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb/volumes"
Sep 09 12:38:07.537336 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:07.537292    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:38:09.362834 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.362763    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4" podUID="88193e18-eff3-4f59-a5dc-29928d04c336" containerName="main" containerID="cri-o://b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49" gracePeriod=45
Sep 09 12:38:09.601488 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.601466    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:38:09.736146 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-home\") pod \"88193e18-eff3-4f59-a5dc-29928d04c336\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") "
Sep 09 12:38:09.736367 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736174    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-model-cache\") pod \"88193e18-eff3-4f59-a5dc-29928d04c336\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") "
Sep 09 12:38:09.736367 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88193e18-eff3-4f59-a5dc-29928d04c336-tls-certs\") pod \"88193e18-eff3-4f59-a5dc-29928d04c336\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") "
Sep 09 12:38:09.736367 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736267    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-tmp-dir\") pod \"88193e18-eff3-4f59-a5dc-29928d04c336\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") "
Sep 09 12:38:09.736367 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736344    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-dshm\") pod \"88193e18-eff3-4f59-a5dc-29928d04c336\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") "
Sep 09 12:38:09.736367 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbjsx\" (UniqueName: \"kubernetes.io/projected/88193e18-eff3-4f59-a5dc-29928d04c336-kube-api-access-qbjsx\") pod \"88193e18-eff3-4f59-a5dc-29928d04c336\" (UID: \"88193e18-eff3-4f59-a5dc-29928d04c336\") "
Sep 09 12:38:09.736644 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736406    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-model-cache" (OuterVolumeSpecName: "model-cache") pod "88193e18-eff3-4f59-a5dc-29928d04c336" (UID: "88193e18-eff3-4f59-a5dc-29928d04c336"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:09.736644 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736453    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-home" (OuterVolumeSpecName: "home") pod "88193e18-eff3-4f59-a5dc-29928d04c336" (UID: "88193e18-eff3-4f59-a5dc-29928d04c336"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:09.736644 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736517    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88193e18-eff3-4f59-a5dc-29928d04c336" (UID: "88193e18-eff3-4f59-a5dc-29928d04c336"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:09.736798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736737    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-home\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:09.736798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736756    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-model-cache\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:09.736798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.736772    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-tmp-dir\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:09.738644 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.738620    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-dshm" (OuterVolumeSpecName: "dshm") pod "88193e18-eff3-4f59-a5dc-29928d04c336" (UID: "88193e18-eff3-4f59-a5dc-29928d04c336"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:09.738723 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.738673    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88193e18-eff3-4f59-a5dc-29928d04c336-kube-api-access-qbjsx" (OuterVolumeSpecName: "kube-api-access-qbjsx") pod "88193e18-eff3-4f59-a5dc-29928d04c336" (UID: "88193e18-eff3-4f59-a5dc-29928d04c336"). InnerVolumeSpecName "kube-api-access-qbjsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:09.738771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.738723    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88193e18-eff3-4f59-a5dc-29928d04c336-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88193e18-eff3-4f59-a5dc-29928d04c336" (UID: "88193e18-eff3-4f59-a5dc-29928d04c336"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:38:09.810128 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.810097    2578 generic.go:356] "Generic (PLEG): container finished" podID="88193e18-eff3-4f59-a5dc-29928d04c336" containerID="b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49" exitCode=0
Sep 09 12:38:09.810325 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.810141    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"
Sep 09 12:38:09.810325 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.810186    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4" event={"ID":"88193e18-eff3-4f59-a5dc-29928d04c336","Type":"ContainerDied","Data":"b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49"}
Sep 09 12:38:09.810325 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.810257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4" event={"ID":"88193e18-eff3-4f59-a5dc-29928d04c336","Type":"ContainerDied","Data":"cfa18f8364ab4e3fa8ae99f6764dc0be4a3a1db064bfdb9340c7935a115c7afc"}
Sep 09 12:38:09.810325 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.810280    2578 scope.go:117] "RemoveContainer" containerID="b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49"
Sep 09 12:38:09.819593 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.819571    2578 scope.go:117] "RemoveContainer" containerID="b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49"
Sep 09 12:38:09.819853 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:09.819837    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49\": container with ID starting with b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49 not found: ID does not exist" containerID="b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49"
Sep 09 12:38:09.819898 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.819862    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49"} err="failed to get container status \"b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49\": rpc error: code = NotFound desc = could not find container \"b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49\": container with ID starting with b03e63b7f0b9d47769ee4a94e549d969ee76c8183494546ae566e3894ee7fc49 not found: ID does not exist"
Sep 09 12:38:09.830738 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.830710    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"]
Sep 09 12:38:09.834379 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.834357    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1e223f/test-model-1e223f-kserve-78bdbb79b7-z2zx4"]
Sep 09 12:38:09.837798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.837778    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88193e18-eff3-4f59-a5dc-29928d04c336-dshm\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:09.837798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.837800    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbjsx\" (UniqueName: \"kubernetes.io/projected/88193e18-eff3-4f59-a5dc-29928d04c336-kube-api-access-qbjsx\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:09.837922 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:09.837809    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88193e18-eff3-4f59-a5dc-29928d04c336-tls-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:10.186375 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:10.186339    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88193e18-eff3-4f59-a5dc-29928d04c336" path="/var/lib/kubelet/pods/88193e18-eff3-4f59-a5dc-29928d04c336/volumes"
Sep 09 12:38:18.734237 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:18.734182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:38:25.828485 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:25.828451    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"]
Sep 09 12:38:25.829023 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:25.828705    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2" podUID="c99afdec-1abf-47e8-8046-c7bbc9db37c7" containerName="maas-api" containerID="cri-o://5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a" gracePeriod=30
Sep 09 12:38:26.079976 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.079907    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:38:26.194307 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.194274    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls\") pod \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") "
Sep 09 12:38:26.194494 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.194313    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-metrics-certs\") pod \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") "
Sep 09 12:38:26.194494 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.194369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99afdec-1abf-47e8-8046-c7bbc9db37c7-openshift-service-ca\") pod \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") "
Sep 09 12:38:26.194701 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.194674    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm9zt\" (UniqueName: \"kubernetes.io/projected/c99afdec-1abf-47e8-8046-c7bbc9db37c7-kube-api-access-sm9zt\") pod \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\" (UID: \"c99afdec-1abf-47e8-8046-c7bbc9db37c7\") "
Sep 09 12:38:26.194853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.194772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99afdec-1abf-47e8-8046-c7bbc9db37c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c99afdec-1abf-47e8-8046-c7bbc9db37c7" (UID: "c99afdec-1abf-47e8-8046-c7bbc9db37c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:38:26.195045 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.195016    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99afdec-1abf-47e8-8046-c7bbc9db37c7-openshift-service-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:26.197248 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.197199    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99afdec-1abf-47e8-8046-c7bbc9db37c7-kube-api-access-sm9zt" (OuterVolumeSpecName: "kube-api-access-sm9zt") pod "c99afdec-1abf-47e8-8046-c7bbc9db37c7" (UID: "c99afdec-1abf-47e8-8046-c7bbc9db37c7"). InnerVolumeSpecName "kube-api-access-sm9zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:26.197398 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.197377    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c99afdec-1abf-47e8-8046-c7bbc9db37c7" (UID: "c99afdec-1abf-47e8-8046-c7bbc9db37c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:38:26.197476 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.197401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c99afdec-1abf-47e8-8046-c7bbc9db37c7" (UID: "c99afdec-1abf-47e8-8046-c7bbc9db37c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:38:26.296549 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.296513    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm9zt\" (UniqueName: \"kubernetes.io/projected/c99afdec-1abf-47e8-8046-c7bbc9db37c7-kube-api-access-sm9zt\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:26.296549 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.296544    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-maas-api-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:26.296549 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.296552    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99afdec-1abf-47e8-8046-c7bbc9db37c7-metrics-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:26.896281 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.896245    2578 generic.go:356] "Generic (PLEG): container finished" podID="c99afdec-1abf-47e8-8046-c7bbc9db37c7" containerID="5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a" exitCode=0
Sep 09 12:38:26.896792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.896346    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"
Sep 09 12:38:26.896792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.896348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2" event={"ID":"c99afdec-1abf-47e8-8046-c7bbc9db37c7","Type":"ContainerDied","Data":"5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a"}
Sep 09 12:38:26.896792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.896405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2" event={"ID":"c99afdec-1abf-47e8-8046-c7bbc9db37c7","Type":"ContainerDied","Data":"7664997f203006f25b34cb93174ff121a8a2dcbc5862334289038d4bb38f949e"}
Sep 09 12:38:26.896792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.896442    2578 scope.go:117] "RemoveContainer" containerID="5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a"
Sep 09 12:38:26.906571 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.906553    2578 scope.go:117] "RemoveContainer" containerID="5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a"
Sep 09 12:38:26.906841 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:26.906822    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a\": container with ID starting with 5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a not found: ID does not exist" containerID="5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a"
Sep 09 12:38:26.906920 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.906854    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a"} err="failed to get container status \"5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a\": rpc error: code = NotFound desc = could not find container \"5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a\": container with ID starting with 5f7ed1689ecdc23f9b9776203962b8bc070d4c370565d4c5773238741589343a not found: ID does not exist"
Sep 09 12:38:26.917026 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.917003    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"]
Sep 09 12:38:26.920935 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:26.920906    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e5923-56df8689c9-2ljf2"]
Sep 09 12:38:28.185685 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:28.185654    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c99afdec-1abf-47e8-8046-c7bbc9db37c7" path="/var/lib/kubelet/pods/c99afdec-1abf-47e8-8046-c7bbc9db37c7/volumes"
Sep 09 12:38:28.226080 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:28.226048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:38:30.825483 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825448    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"]
Sep 09 12:38:30.825862 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825849    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb" containerName="payload-processing"
Sep 09 12:38:30.825862 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825860    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb" containerName="payload-processing"
Sep 09 12:38:30.825937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825879    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21841879-125f-4030-ba71-8accec6be937" containerName="payload-pre-processing"
Sep 09 12:38:30.825937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825884    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21841879-125f-4030-ba71-8accec6be937" containerName="payload-pre-processing"
Sep 09 12:38:30.825937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825898    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c99afdec-1abf-47e8-8046-c7bbc9db37c7" containerName="maas-api"
Sep 09 12:38:30.825937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825904    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c99afdec-1abf-47e8-8046-c7bbc9db37c7" containerName="maas-api"
Sep 09 12:38:30.825937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825914    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07cbcacf-e0ef-4629-908a-984ba481f064" containerName="maas-api"
Sep 09 12:38:30.825937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825919    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="07cbcacf-e0ef-4629-908a-984ba481f064" containerName="maas-api"
Sep 09 12:38:30.825937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825927    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88193e18-eff3-4f59-a5dc-29928d04c336" containerName="main"
Sep 09 12:38:30.825937 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825932    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88193e18-eff3-4f59-a5dc-29928d04c336" containerName="main"
Sep 09 12:38:30.826178 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825984    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd8346db-d0ca-4c1f-88cb-ceb1c9fa5afb" containerName="payload-processing"
Sep 09 12:38:30.826178 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.825994    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c99afdec-1abf-47e8-8046-c7bbc9db37c7" containerName="maas-api"
Sep 09 12:38:30.826178 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.826003    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88193e18-eff3-4f59-a5dc-29928d04c336" containerName="main"
Sep 09 12:38:30.826178 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.826009    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="07cbcacf-e0ef-4629-908a-984ba481f064" containerName="maas-api"
Sep 09 12:38:30.826178 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.826014    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21841879-125f-4030-ba71-8accec6be937" containerName="payload-pre-processing"
Sep 09 12:38:30.830587 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.830572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:30.833492 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.833459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6a8ecc\"/\"default-dockercfg-jqdkx\""
Sep 09 12:38:30.833492 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.833472    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6a8ecc\"/\"kube-root-ca.crt\""
Sep 09 12:38:30.833661 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.833518    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6a8ecc\"/\"openshift-service-ca.crt\""
Sep 09 12:38:30.833661 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.833518    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6a8ecc\"/\"noten-model-6a8ecc-kserve-self-signed-certs\""
Sep 09 12:38:30.840099 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.840078    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"]
Sep 09 12:38:30.940587 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.940550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-model-cache\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:30.940587 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.940587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw9b7\" (UniqueName: \"kubernetes.io/projected/cca520d0-217c-46c7-883c-a0ef813c9fc8-kube-api-access-zw9b7\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:30.940806 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.940663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-tmp-dir\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:30.940806 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.940745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-home\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:30.940806 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.940785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-dshm\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:30.940908 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:30.940810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.041500 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.041461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-model-cache\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.041500 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.041501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw9b7\" (UniqueName: \"kubernetes.io/projected/cca520d0-217c-46c7-883c-a0ef813c9fc8-kube-api-access-zw9b7\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.041742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.041555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-tmp-dir\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.041742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.041605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-home\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.041742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.041629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-dshm\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.041742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.041648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.041930 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.041893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-model-cache\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.041930 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.041915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-tmp-dir\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.042017 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.041997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-home\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.043979 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.043958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-dshm\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.044200 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.044184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.049505 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.049487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw9b7\" (UniqueName: \"kubernetes.io/projected/cca520d0-217c-46c7-883c-a0ef813c9fc8-kube-api-access-zw9b7\") pod \"noten-model-6a8ecc-kserve-769944d68c-h2wv6\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") " pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.140633 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.140552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:31.412457 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.412433    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"]
Sep 09 12:38:31.414633 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:38:31.414608    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcca520d0_217c_46c7_883c_a0ef813c9fc8.slice/crio-760f87d74d29ae23313adcb90d865e4b4d74402c031768d867010b85dab1c511 WatchSource:0}: Error finding container 760f87d74d29ae23313adcb90d865e4b4d74402c031768d867010b85dab1c511: Status 404 returned error can't find the container with id 760f87d74d29ae23313adcb90d865e4b4d74402c031768d867010b85dab1c511
Sep 09 12:38:31.923310 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.923274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6" event={"ID":"cca520d0-217c-46c7-883c-a0ef813c9fc8","Type":"ContainerStarted","Data":"7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd"}
Sep 09 12:38:31.923310 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.923315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6" event={"ID":"cca520d0-217c-46c7-883c-a0ef813c9fc8","Type":"ContainerStarted","Data":"760f87d74d29ae23313adcb90d865e4b4d74402c031768d867010b85dab1c511"}
Sep 09 12:38:31.952978 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:31.952931    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6" podStartSLOduration=1.952915678 podStartE2EDuration="1.952915678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:38:31.950725031 +0000 UTC m=+1132.294322978" watchObservedRunningTime="2026-09-09 12:38:31.952915678 +0000 UTC m=+1132.296513626"
Sep 09 12:38:39.690556 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:39.690519    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"]
Sep 09 12:38:39.694404 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:39.694382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:38:39.696557 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:39.696530    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-283adb\""
Sep 09 12:38:39.696697 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:39.696632    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-283adb-dockercfg-v2ghh\""
Sep 09 12:38:39.703951 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:39.703924    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"]
Sep 09 12:38:39.827999 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:39.827966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7h4l\" (UniqueName: \"kubernetes.io/projected/4256461e-70dc-436f-84bf-707057b5e1a8-kube-api-access-l7h4l\") pod \"payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx\" (UID: \"4256461e-70dc-436f-84bf-707057b5e1a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:38:39.828166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:39.828018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4256461e-70dc-436f-84bf-707057b5e1a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx\" (UID: \"4256461e-70dc-436f-84bf-707057b5e1a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:38:39.929192 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:39.929160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7h4l\" (UniqueName: \"kubernetes.io/projected/4256461e-70dc-436f-84bf-707057b5e1a8-kube-api-access-l7h4l\") pod \"payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx\" (UID: \"4256461e-70dc-436f-84bf-707057b5e1a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:38:39.929365 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:39.929209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4256461e-70dc-436f-84bf-707057b5e1a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx\" (UID: \"4256461e-70dc-436f-84bf-707057b5e1a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:38:39.929901 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:39.929881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4256461e-70dc-436f-84bf-707057b5e1a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx\" (UID: \"4256461e-70dc-436f-84bf-707057b5e1a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:38:39.938619 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:39.938593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7h4l\" (UniqueName: \"kubernetes.io/projected/4256461e-70dc-436f-84bf-707057b5e1a8-kube-api-access-l7h4l\") pod \"payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx\" (UID: \"4256461e-70dc-436f-84bf-707057b5e1a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:38:40.005899 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:40.005819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:38:40.281270 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:40.281172    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"]
Sep 09 12:38:40.283000 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:38:40.282977    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4256461e_70dc_436f_84bf_707057b5e1a8.slice/crio-b73aa6bac1a2606b6adb2b480e8bd86da415c96adab79cae2db6d51a1356877d WatchSource:0}: Error finding container b73aa6bac1a2606b6adb2b480e8bd86da415c96adab79cae2db6d51a1356877d: Status 404 returned error can't find the container with id b73aa6bac1a2606b6adb2b480e8bd86da415c96adab79cae2db6d51a1356877d
Sep 09 12:38:40.972678 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:40.972629    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx" event={"ID":"4256461e-70dc-436f-84bf-707057b5e1a8","Type":"ContainerStarted","Data":"b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3"}
Sep 09 12:38:40.973064 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:40.972693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx" event={"ID":"4256461e-70dc-436f-84bf-707057b5e1a8","Type":"ContainerStarted","Data":"b73aa6bac1a2606b6adb2b480e8bd86da415c96adab79cae2db6d51a1356877d"}
Sep 09 12:38:40.973064 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:40.972722    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:38:40.997731 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:40.997673    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx" podStartSLOduration=1.997657558 podStartE2EDuration="1.997657558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:38:40.995929647 +0000 UTC m=+1141.339527597" watchObservedRunningTime="2026-09-09 12:38:40.997657558 +0000 UTC m=+1141.341255506"
Sep 09 12:38:41.141192 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:41.141154    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:41.141386 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:41.141234    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:41.153535 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:41.153506    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:41.989041 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:41.989016    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:42.914243 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:42.914198    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"]
Sep 09 12:38:43.061234 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:43.061188    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-self-signed-certs: secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:43.061612 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:43.061304    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs podName:cca520d0-217c-46c7-883c-a0ef813c9fc8 nodeName:}" failed. No retries permitted until 2026-09-09 12:38:43.561279176 +0000 UTC m=+1143.904877116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs") pod "noten-model-6a8ecc-kserve-769944d68c-h2wv6" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8") : secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:43.127173 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:43.127138    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:38:43.566427 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:43.566387    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-self-signed-certs: secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:43.566606 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:43.566459    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs podName:cca520d0-217c-46c7-883c-a0ef813c9fc8 nodeName:}" failed. No retries permitted until 2026-09-09 12:38:44.566443844 +0000 UTC m=+1144.910041769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs") pod "noten-model-6a8ecc-kserve-769944d68c-h2wv6" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8") : secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:44.576729 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:44.576686    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-self-signed-certs: secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:44.577196 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:44.576770    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs podName:cca520d0-217c-46c7-883c-a0ef813c9fc8 nodeName:}" failed. No retries permitted until 2026-09-09 12:38:46.576752172 +0000 UTC m=+1146.920350098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs") pod "noten-model-6a8ecc-kserve-769944d68c-h2wv6" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8") : secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:45.113516 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.113477    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"]
Sep 09 12:38:45.117290 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.117274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:38:45.119092 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.119070    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a3f43f\""
Sep 09 12:38:45.119348 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.119335    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a3f43f-dockercfg-cptxz\""
Sep 09 12:38:45.125657 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.125628    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"]
Sep 09 12:38:45.182809 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.182780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwgjd\" (UniqueName: \"kubernetes.io/projected/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-kube-api-access-nwgjd\") pod \"payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d\" (UID: \"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:38:45.182967 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.182829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d\" (UID: \"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:38:45.283943 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.283908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwgjd\" (UniqueName: \"kubernetes.io/projected/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-kube-api-access-nwgjd\") pod \"payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d\" (UID: \"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:38:45.284141 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.283961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d\" (UID: \"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:38:45.284757 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.284733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d\" (UID: \"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:38:45.292069 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.292050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwgjd\" (UniqueName: \"kubernetes.io/projected/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-kube-api-access-nwgjd\") pod \"payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d\" (UID: \"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:38:45.429269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.429228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:38:45.701460 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.701430    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"]
Sep 09 12:38:45.704096 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:38:45.704067    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92ca333_f4d8_4d40_a6e9_33cefcccb2b2.slice/crio-f2f652010c88da48b6f5f4e7e3d229a3530bf94a3606b498fd9987b8eecbcd9e WatchSource:0}: Error finding container f2f652010c88da48b6f5f4e7e3d229a3530bf94a3606b498fd9987b8eecbcd9e: Status 404 returned error can't find the container with id f2f652010c88da48b6f5f4e7e3d229a3530bf94a3606b498fd9987b8eecbcd9e
Sep 09 12:38:45.997382 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.997292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d" event={"ID":"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2","Type":"ContainerStarted","Data":"e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b"}
Sep 09 12:38:45.997382 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.997350    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d" event={"ID":"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2","Type":"ContainerStarted","Data":"f2f652010c88da48b6f5f4e7e3d229a3530bf94a3606b498fd9987b8eecbcd9e"}
Sep 09 12:38:45.997613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:45.997467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:38:46.015708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:46.015635    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d" podStartSLOduration=1.015614429 podStartE2EDuration="1.015614429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:38:46.013365279 +0000 UTC m=+1146.356963240" watchObservedRunningTime="2026-09-09 12:38:46.015614429 +0000 UTC m=+1146.359212378"
Sep 09 12:38:46.598498 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:46.598464    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-self-signed-certs: secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:46.598687 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:46.598542    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs podName:cca520d0-217c-46c7-883c-a0ef813c9fc8 nodeName:}" failed. No retries permitted until 2026-09-09 12:38:50.598523815 +0000 UTC m=+1150.942121742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs") pod "noten-model-6a8ecc-kserve-769944d68c-h2wv6" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8") : secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:50.640762 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:50.640725    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-self-signed-certs: secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:50.641257 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:50.640804    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs podName:cca520d0-217c-46c7-883c-a0ef813c9fc8 nodeName:}" failed. No retries permitted until 2026-09-09 12:38:58.640786702 +0000 UTC m=+1158.984384627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs") pod "noten-model-6a8ecc-kserve-769944d68c-h2wv6" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8") : secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:51.979464 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:51.979437    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:38:52.840181 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:52.840145    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:38:57.005279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:57.005249    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:38:58.723584 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:58.723546    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-self-signed-certs: secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:58.723966 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:38:58.723621    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs podName:cca520d0-217c-46c7-883c-a0ef813c9fc8 nodeName:}" failed. No retries permitted until 2026-09-09 12:39:14.723604196 +0000 UTC m=+1175.067202125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs") pod "noten-model-6a8ecc-kserve-769944d68c-h2wv6" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8") : secret "noten-model-6a8ecc-kserve-self-signed-certs" not found
Sep 09 12:38:59.006611 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.006522    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6" podUID="cca520d0-217c-46c7-883c-a0ef813c9fc8" containerName="main" containerID="cri-o://7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd" gracePeriod=45
Sep 09 12:38:59.253434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.253406    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:38:59.331293 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.331164    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs\") pod \"cca520d0-217c-46c7-883c-a0ef813c9fc8\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") "
Sep 09 12:38:59.331293 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.331241    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-model-cache\") pod \"cca520d0-217c-46c7-883c-a0ef813c9fc8\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") "
Sep 09 12:38:59.331293 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.331287    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-tmp-dir\") pod \"cca520d0-217c-46c7-883c-a0ef813c9fc8\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") "
Sep 09 12:38:59.331603 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.331313    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-home\") pod \"cca520d0-217c-46c7-883c-a0ef813c9fc8\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") "
Sep 09 12:38:59.331603 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.331338    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-dshm\") pod \"cca520d0-217c-46c7-883c-a0ef813c9fc8\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") "
Sep 09 12:38:59.331603 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.331356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw9b7\" (UniqueName: \"kubernetes.io/projected/cca520d0-217c-46c7-883c-a0ef813c9fc8-kube-api-access-zw9b7\") pod \"cca520d0-217c-46c7-883c-a0ef813c9fc8\" (UID: \"cca520d0-217c-46c7-883c-a0ef813c9fc8\") "
Sep 09 12:38:59.331603 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.331569    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-model-cache" (OuterVolumeSpecName: "model-cache") pod "cca520d0-217c-46c7-883c-a0ef813c9fc8" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:59.331775 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.331622    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-home" (OuterVolumeSpecName: "home") pod "cca520d0-217c-46c7-883c-a0ef813c9fc8" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:59.331775 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.331633    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cca520d0-217c-46c7-883c-a0ef813c9fc8" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:59.333731 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.333705    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-dshm" (OuterVolumeSpecName: "dshm") pod "cca520d0-217c-46c7-883c-a0ef813c9fc8" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:59.333827 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.333769    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cca520d0-217c-46c7-883c-a0ef813c9fc8" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:38:59.333827 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.333778    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca520d0-217c-46c7-883c-a0ef813c9fc8-kube-api-access-zw9b7" (OuterVolumeSpecName: "kube-api-access-zw9b7") pod "cca520d0-217c-46c7-883c-a0ef813c9fc8" (UID: "cca520d0-217c-46c7-883c-a0ef813c9fc8"). InnerVolumeSpecName "kube-api-access-zw9b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:59.432368 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.432324    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-tmp-dir\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.432368 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.432359    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-home\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.432368 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.432371    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-dshm\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.432620 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.432384    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw9b7\" (UniqueName: \"kubernetes.io/projected/cca520d0-217c-46c7-883c-a0ef813c9fc8-kube-api-access-zw9b7\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.432620 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.432399    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca520d0-217c-46c7-883c-a0ef813c9fc8-tls-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.432620 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:38:59.432409    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca520d0-217c-46c7-883c-a0ef813c9fc8-model-cache\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:39:00.078619 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:00.078585    2578 generic.go:356] "Generic (PLEG): container finished" podID="cca520d0-217c-46c7-883c-a0ef813c9fc8" containerID="7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd" exitCode=0
Sep 09 12:39:00.079040 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:00.078625    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"
Sep 09 12:39:00.079040 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:00.078640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6" event={"ID":"cca520d0-217c-46c7-883c-a0ef813c9fc8","Type":"ContainerDied","Data":"7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd"}
Sep 09 12:39:00.079040 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:00.078679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6" event={"ID":"cca520d0-217c-46c7-883c-a0ef813c9fc8","Type":"ContainerDied","Data":"760f87d74d29ae23313adcb90d865e4b4d74402c031768d867010b85dab1c511"}
Sep 09 12:39:00.079040 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:00.078697    2578 scope.go:117] "RemoveContainer" containerID="7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd"
Sep 09 12:39:00.087986 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:00.087963    2578 scope.go:117] "RemoveContainer" containerID="7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd"
Sep 09 12:39:00.088335 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:39:00.088313    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd\": container with ID starting with 7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd not found: ID does not exist" containerID="7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd"
Sep 09 12:39:00.088395 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:00.088344    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd"} err="failed to get container status \"7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd\": rpc error: code = NotFound desc = could not find container \"7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd\": container with ID starting with 7c175d89f830c4bdfd2a28f38ae8d2cb74f0f957272b1c1587f48548f2d955bd not found: ID does not exist"
Sep 09 12:39:00.099986 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:00.099960    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"]
Sep 09 12:39:00.101637 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:00.101618    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-6a8ecc/noten-model-6a8ecc-kserve-769944d68c-h2wv6"]
Sep 09 12:39:00.188193 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:00.188161    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cca520d0-217c-46c7-883c-a0ef813c9fc8" path="/var/lib/kubelet/pods/cca520d0-217c-46c7-883c-a0ef813c9fc8/volumes"
Sep 09 12:39:07.629169 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.629083    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"]
Sep 09 12:39:07.629774 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.629754    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca520d0-217c-46c7-883c-a0ef813c9fc8" containerName="main"
Sep 09 12:39:07.629824 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.629780    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca520d0-217c-46c7-883c-a0ef813c9fc8" containerName="main"
Sep 09 12:39:07.629936 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.629923    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca520d0-217c-46c7-883c-a0ef813c9fc8" containerName="main"
Sep 09 12:39:07.633470 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.633448    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.635501 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.635472    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-fe6606-serving-cert\""
Sep 09 12:39:07.635630 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.635585    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 12:39:07.636302 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.636285    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 12:39:07.636302 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.636297    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 12:39:07.636453 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.636320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5rdxz\""
Sep 09 12:39:07.640764 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.640735    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"]
Sep 09 12:39:07.711856 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.711826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.712032 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.711864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vfw\" (UniqueName: \"kubernetes.io/projected/1de12030-3db6-45d3-8349-e9d856c1892a-kube-api-access-m4vfw\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.712032 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.711889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de12030-3db6-45d3-8349-e9d856c1892a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.712032 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.711918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.812702 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.812669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.812868 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.812712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vfw\" (UniqueName: \"kubernetes.io/projected/1de12030-3db6-45d3-8349-e9d856c1892a-kube-api-access-m4vfw\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.812868 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.812734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de12030-3db6-45d3-8349-e9d856c1892a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.812868 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.812766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.813528 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.813505    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de12030-3db6-45d3-8349-e9d856c1892a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.815409 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.815387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.815522 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.815387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.820491 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.820466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vfw\" (UniqueName: \"kubernetes.io/projected/1de12030-3db6-45d3-8349-e9d856c1892a-kube-api-access-m4vfw\") pod \"maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:07.948093 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:07.948061    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:08.225634 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.225609    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"]
Sep 09 12:39:08.228071 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:39:08.228041    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de12030_3db6_45d3_8349_e9d856c1892a.slice/crio-5f1ac525ecb0bde111ef98173290e66e1d98e0f12544777e1daa37a259977a49 WatchSource:0}: Error finding container 5f1ac525ecb0bde111ef98173290e66e1d98e0f12544777e1daa37a259977a49: Status 404 returned error can't find the container with id 5f1ac525ecb0bde111ef98173290e66e1d98e0f12544777e1daa37a259977a49
Sep 09 12:39:08.296855 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.296822    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"]
Sep 09 12:39:08.301398 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.301383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:39:08.303862 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.303839    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-fe6606\""
Sep 09 12:39:08.304009 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.303909    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-fe6606-dockercfg-7gp5r\""
Sep 09 12:39:08.311719 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.311698    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"]
Sep 09 12:39:08.418649 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.418617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fddfae9-dfa3-487d-910a-5b3400648c38-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6\" (UID: \"0fddfae9-dfa3-487d-910a-5b3400648c38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:39:08.418837 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.418703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4v7\" (UniqueName: \"kubernetes.io/projected/0fddfae9-dfa3-487d-910a-5b3400648c38-kube-api-access-cp4v7\") pod \"payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6\" (UID: \"0fddfae9-dfa3-487d-910a-5b3400648c38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:39:08.520064 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.519976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fddfae9-dfa3-487d-910a-5b3400648c38-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6\" (UID: \"0fddfae9-dfa3-487d-910a-5b3400648c38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:39:08.520242 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.520063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4v7\" (UniqueName: \"kubernetes.io/projected/0fddfae9-dfa3-487d-910a-5b3400648c38-kube-api-access-cp4v7\") pod \"payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6\" (UID: \"0fddfae9-dfa3-487d-910a-5b3400648c38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:39:08.520712 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.520693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fddfae9-dfa3-487d-910a-5b3400648c38-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6\" (UID: \"0fddfae9-dfa3-487d-910a-5b3400648c38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:39:08.528250 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.528234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4v7\" (UniqueName: \"kubernetes.io/projected/0fddfae9-dfa3-487d-910a-5b3400648c38-kube-api-access-cp4v7\") pod \"payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6\" (UID: \"0fddfae9-dfa3-487d-910a-5b3400648c38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:39:08.613601 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.613564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:39:08.890336 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:08.890309    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"]
Sep 09 12:39:08.892606 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:39:08.892576    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fddfae9_dfa3_487d_910a_5b3400648c38.slice/crio-c8efbca4d584dd0000f7b18713884224ef8ff1586e1010bc7dc30987ba0bf7d8 WatchSource:0}: Error finding container c8efbca4d584dd0000f7b18713884224ef8ff1586e1010bc7dc30987ba0bf7d8: Status 404 returned error can't find the container with id c8efbca4d584dd0000f7b18713884224ef8ff1586e1010bc7dc30987ba0bf7d8
Sep 09 12:39:09.130126 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:09.130086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t" event={"ID":"1de12030-3db6-45d3-8349-e9d856c1892a","Type":"ContainerStarted","Data":"5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4"}
Sep 09 12:39:09.130360 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:09.130151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:09.130360 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:09.130175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t" event={"ID":"1de12030-3db6-45d3-8349-e9d856c1892a","Type":"ContainerStarted","Data":"5f1ac525ecb0bde111ef98173290e66e1d98e0f12544777e1daa37a259977a49"}
Sep 09 12:39:09.132176 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:09.131901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6" event={"ID":"0fddfae9-dfa3-487d-910a-5b3400648c38","Type":"ContainerStarted","Data":"b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1"}
Sep 09 12:39:09.132176 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:09.131952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6" event={"ID":"0fddfae9-dfa3-487d-910a-5b3400648c38","Type":"ContainerStarted","Data":"c8efbca4d584dd0000f7b18713884224ef8ff1586e1010bc7dc30987ba0bf7d8"}
Sep 09 12:39:09.132176 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:09.132176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:39:09.149079 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:09.148996    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t" podStartSLOduration=1.7639789270000001 podStartE2EDuration="2.148981931s" totalImagesPullingTime="385.003004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:39:09.147874712 +0000 UTC m=+1169.491472661" watchObservedRunningTime="2026-09-09 12:39:09.148981931 +0000 UTC m=+1169.492579879"
Sep 09 12:39:09.164815 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:09.164753    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6" podStartSLOduration=1.164737094 podStartE2EDuration="1.164737094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:39:09.163903033 +0000 UTC m=+1169.507500982" watchObservedRunningTime="2026-09-09 12:39:09.164737094 +0000 UTC m=+1169.508335046"
Sep 09 12:39:12.638938 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:12.638907    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:39:14.052928 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.052889    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"]
Sep 09 12:39:14.057559 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.057538    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:39:14.059575 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.059555    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-590823-dockercfg-xtfnd\""
Sep 09 12:39:14.059689 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.059582    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-590823\""
Sep 09 12:39:14.064634 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.064608    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"]
Sep 09 12:39:14.170547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.170513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj\" (UID: \"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:39:14.170721 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.170626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpzd8\" (UniqueName: \"kubernetes.io/projected/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-kube-api-access-zpzd8\") pod \"payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj\" (UID: \"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:39:14.271806 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.271771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj\" (UID: \"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:39:14.271973 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.271864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzd8\" (UniqueName: \"kubernetes.io/projected/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-kube-api-access-zpzd8\") pod \"payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj\" (UID: \"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:39:14.272502 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.272481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj\" (UID: \"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:39:14.280171 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.280146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzd8\" (UniqueName: \"kubernetes.io/projected/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-kube-api-access-zpzd8\") pod \"payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj\" (UID: \"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:39:14.369241 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.369135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:39:14.641272 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:14.641246    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"]
Sep 09 12:39:14.643862 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:39:14.643829    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8e5a13_9f30_4c93_b9ea_e24c1e54a641.slice/crio-a6b7d4e5a7ff1bd3c9b45ce57c14b6cde8278050a336ba86424a1f96d83d76e7 WatchSource:0}: Error finding container a6b7d4e5a7ff1bd3c9b45ce57c14b6cde8278050a336ba86424a1f96d83d76e7: Status 404 returned error can't find the container with id a6b7d4e5a7ff1bd3c9b45ce57c14b6cde8278050a336ba86424a1f96d83d76e7
Sep 09 12:39:15.143472 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:15.143435    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:39:15.174326 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:15.174267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj" event={"ID":"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641","Type":"ContainerStarted","Data":"1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8"}
Sep 09 12:39:15.174517 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:15.174396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj" event={"ID":"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641","Type":"ContainerStarted","Data":"a6b7d4e5a7ff1bd3c9b45ce57c14b6cde8278050a336ba86424a1f96d83d76e7"}
Sep 09 12:39:15.174594 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:15.174559    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:39:15.191123 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:15.191075    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj" podStartSLOduration=1.1910599610000001 podStartE2EDuration="1.191059961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:39:15.188913753 +0000 UTC m=+1175.532511704" watchObservedRunningTime="2026-09-09 12:39:15.191059961 +0000 UTC m=+1175.534657909"
Sep 09 12:39:20.140419 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:20.140387    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:39:26.186304 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:26.186274    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:39:28.133860 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:28.133812    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:39:37.877524 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:37.877490    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7672-15555a"]
Sep 09 12:39:37.881279 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:37.881262    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-15555a"
Sep 09 12:39:37.883604 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:37.883584    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4dspl\""
Sep 09 12:39:37.888188 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:37.888167    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7672-15555a"]
Sep 09 12:39:37.904039 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:37.904013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qvt\" (UniqueName: \"kubernetes.io/projected/fadcf98a-5570-4cea-b78a-875633def6f5-kube-api-access-75qvt\") pod \"test-curl-7672-15555a\" (UID: \"fadcf98a-5570-4cea-b78a-875633def6f5\") " pod="opendatahub/test-curl-7672-15555a"
Sep 09 12:39:38.005495 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:38.005463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qvt\" (UniqueName: \"kubernetes.io/projected/fadcf98a-5570-4cea-b78a-875633def6f5-kube-api-access-75qvt\") pod \"test-curl-7672-15555a\" (UID: \"fadcf98a-5570-4cea-b78a-875633def6f5\") " pod="opendatahub/test-curl-7672-15555a"
Sep 09 12:39:38.013581 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:38.013555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qvt\" (UniqueName: \"kubernetes.io/projected/fadcf98a-5570-4cea-b78a-875633def6f5-kube-api-access-75qvt\") pod \"test-curl-7672-15555a\" (UID: \"fadcf98a-5570-4cea-b78a-875633def6f5\") " pod="opendatahub/test-curl-7672-15555a"
Sep 09 12:39:38.192791 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:38.192768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-15555a"
Sep 09 12:39:38.469095 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:38.469070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7672-15555a"]
Sep 09 12:39:38.471167 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:39:38.471141    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfadcf98a_5570_4cea_b78a_875633def6f5.slice/crio-8d3053def1ebe8978d3600e6708b078992a9d85369d5d6e016c7df767bbca221 WatchSource:0}: Error finding container 8d3053def1ebe8978d3600e6708b078992a9d85369d5d6e016c7df767bbca221: Status 404 returned error can't find the container with id 8d3053def1ebe8978d3600e6708b078992a9d85369d5d6e016c7df767bbca221
Sep 09 12:39:39.303618 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:39.303581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7672-15555a" event={"ID":"fadcf98a-5570-4cea-b78a-875633def6f5","Type":"ContainerStarted","Data":"a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4"}
Sep 09 12:39:39.303618 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:39.303624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7672-15555a" event={"ID":"fadcf98a-5570-4cea-b78a-875633def6f5","Type":"ContainerStarted","Data":"8d3053def1ebe8978d3600e6708b078992a9d85369d5d6e016c7df767bbca221"}
Sep 09 12:39:39.318716 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:39.318657    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7672-15555a" podStartSLOduration=2.318641009 podStartE2EDuration="2.318641009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:39:39.316577815 +0000 UTC m=+1199.660175764" watchObservedRunningTime="2026-09-09 12:39:39.318641009 +0000 UTC m=+1199.662238958"
Sep 09 12:39:39.912582 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:39.912543    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7672-15555a"]
Sep 09 12:39:39.915070 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:39.915043    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7672-15555a"]
Sep 09 12:39:40.032003 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.031921    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7672-281116"]
Sep 09 12:39:40.032402 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.032389    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fadcf98a-5570-4cea-b78a-875633def6f5" containerName="test-curl-7672-15555a"
Sep 09 12:39:40.032452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.032404    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fadcf98a-5570-4cea-b78a-875633def6f5" containerName="test-curl-7672-15555a"
Sep 09 12:39:40.032487 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.032465    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fadcf98a-5570-4cea-b78a-875633def6f5" containerName="test-curl-7672-15555a"
Sep 09 12:39:40.035692 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.035675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-281116"
Sep 09 12:39:40.041482 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.041458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7672-281116"]
Sep 09 12:39:40.127274 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.127243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zlgq\" (UniqueName: \"kubernetes.io/projected/a4694532-34f0-478f-a92a-9e0f8c6415b4-kube-api-access-6zlgq\") pod \"test-curl-7672-281116\" (UID: \"a4694532-34f0-478f-a92a-9e0f8c6415b4\") " pod="opendatahub/test-curl-7672-281116"
Sep 09 12:39:40.218278 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.218251    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:39:40.222109 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.222085    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:39:40.227961 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.227933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zlgq\" (UniqueName: \"kubernetes.io/projected/a4694532-34f0-478f-a92a-9e0f8c6415b4-kube-api-access-6zlgq\") pod \"test-curl-7672-281116\" (UID: \"a4694532-34f0-478f-a92a-9e0f8c6415b4\") " pod="opendatahub/test-curl-7672-281116"
Sep 09 12:39:40.238134 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.238108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zlgq\" (UniqueName: \"kubernetes.io/projected/a4694532-34f0-478f-a92a-9e0f8c6415b4-kube-api-access-6zlgq\") pod \"test-curl-7672-281116\" (UID: \"a4694532-34f0-478f-a92a-9e0f8c6415b4\") " pod="opendatahub/test-curl-7672-281116"
Sep 09 12:39:40.308944 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:39:40.308923    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7672-15555a" container="test-curl-7672-15555a"
Sep 09 12:39:40.347233 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.347181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-281116"
Sep 09 12:39:40.619908 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:40.619880    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7672-281116"]
Sep 09 12:39:40.621920 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:39:40.621895    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4694532_34f0_478f_a92a_9e0f8c6415b4.slice/crio-0426bd482d7e8089c0937c33cca807ab9f1140fb5efcfdb3406190fcb6159bec WatchSource:0}: Error finding container 0426bd482d7e8089c0937c33cca807ab9f1140fb5efcfdb3406190fcb6159bec: Status 404 returned error can't find the container with id 0426bd482d7e8089c0937c33cca807ab9f1140fb5efcfdb3406190fcb6159bec
Sep 09 12:39:41.315899 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:41.315865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7672-281116" event={"ID":"a4694532-34f0-478f-a92a-9e0f8c6415b4","Type":"ContainerStarted","Data":"4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db"}
Sep 09 12:39:41.315899 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:41.315906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7672-281116" event={"ID":"a4694532-34f0-478f-a92a-9e0f8c6415b4","Type":"ContainerStarted","Data":"0426bd482d7e8089c0937c33cca807ab9f1140fb5efcfdb3406190fcb6159bec"}
Sep 09 12:39:41.316425 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:41.315992    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7672-15555a" podUID="fadcf98a-5570-4cea-b78a-875633def6f5" containerName="test-curl-7672-15555a" containerID="cri-o://a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4" gracePeriod=2
Sep 09 12:39:41.335368 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:41.335285    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7672-281116" podStartSLOduration=1.335266123 podStartE2EDuration="1.335266123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:39:41.3348852 +0000 UTC m=+1201.678483144" watchObservedRunningTime="2026-09-09 12:39:41.335266123 +0000 UTC m=+1201.678864072"
Sep 09 12:39:41.924593 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:41.924563    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7672-281116"]
Sep 09 12:39:41.930269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:41.930201    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:39:41.932257 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:41.932232    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7672-281116"]
Sep 09 12:39:42.047041 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:42.046971    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7672-d4481a"]
Sep 09 12:39:42.047427 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:42.047414    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4694532-34f0-478f-a92a-9e0f8c6415b4" containerName="test-curl-7672-281116"
Sep 09 12:39:42.047493 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:42.047428    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4694532-34f0-478f-a92a-9e0f8c6415b4" containerName="test-curl-7672-281116"
Sep 09 12:39:42.047531 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:42.047500    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4694532-34f0-478f-a92a-9e0f8c6415b4" containerName="test-curl-7672-281116"
Sep 09 12:39:42.050709 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:42.050694    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-d4481a"
Sep 09 12:39:42.057480 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:42.057457    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7672-d4481a"]
Sep 09 12:39:42.144157 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:42.144120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfb27\" (UniqueName: \"kubernetes.io/projected/7724f38b-7333-4043-b67b-188ae27e9841-kube-api-access-wfb27\") pod \"test-curl-7672-d4481a\" (UID: \"7724f38b-7333-4043-b67b-188ae27e9841\") " pod="opendatahub/test-curl-7672-d4481a"
Sep 09 12:39:42.244836 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:42.244808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfb27\" (UniqueName: \"kubernetes.io/projected/7724f38b-7333-4043-b67b-188ae27e9841-kube-api-access-wfb27\") pod \"test-curl-7672-d4481a\" (UID: \"7724f38b-7333-4043-b67b-188ae27e9841\") " pod="opendatahub/test-curl-7672-d4481a"
Sep 09 12:39:42.253662 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:42.253632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfb27\" (UniqueName: \"kubernetes.io/projected/7724f38b-7333-4043-b67b-188ae27e9841-kube-api-access-wfb27\") pod \"test-curl-7672-d4481a\" (UID: \"7724f38b-7333-4043-b67b-188ae27e9841\") " pod="opendatahub/test-curl-7672-d4481a"
Sep 09 12:39:42.321015 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:39:42.320944    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7672-281116" container="test-curl-7672-281116"
Sep 09 12:39:42.360985 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:42.360960    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-d4481a"
Sep 09 12:39:42.629100 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:42.629081    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7672-d4481a"]
Sep 09 12:39:42.631368 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:39:42.631341    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7724f38b_7333_4043_b67b_188ae27e9841.slice/crio-8df3ed3a8a205847e8c0b73ded0730fe7e13813b786e90a6b9a6fd525416ba05 WatchSource:0}: Error finding container 8df3ed3a8a205847e8c0b73ded0730fe7e13813b786e90a6b9a6fd525416ba05: Status 404 returned error can't find the container with id 8df3ed3a8a205847e8c0b73ded0730fe7e13813b786e90a6b9a6fd525416ba05
Sep 09 12:39:43.327777 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:43.327745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7672-d4481a" event={"ID":"7724f38b-7333-4043-b67b-188ae27e9841","Type":"ContainerStarted","Data":"930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22"}
Sep 09 12:39:43.328112 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:43.327797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7672-d4481a" event={"ID":"7724f38b-7333-4043-b67b-188ae27e9841","Type":"ContainerStarted","Data":"8df3ed3a8a205847e8c0b73ded0730fe7e13813b786e90a6b9a6fd525416ba05"}
Sep 09 12:39:43.328112 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:43.327918    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7672-281116" podUID="a4694532-34f0-478f-a92a-9e0f8c6415b4" containerName="test-curl-7672-281116" containerID="cri-o://4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db" gracePeriod=2
Sep 09 12:39:43.343408 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:43.343365    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7672-d4481a" podStartSLOduration=1.343349269 podStartE2EDuration="1.343349269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:39:43.34111494 +0000 UTC m=+1203.684712889" watchObservedRunningTime="2026-09-09 12:39:43.343349269 +0000 UTC m=+1203.686947254"
Sep 09 12:39:43.499365 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:43.499342    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-15555a"
Sep 09 12:39:43.559022 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:43.558995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75qvt\" (UniqueName: \"kubernetes.io/projected/fadcf98a-5570-4cea-b78a-875633def6f5-kube-api-access-75qvt\") pod \"fadcf98a-5570-4cea-b78a-875633def6f5\" (UID: \"fadcf98a-5570-4cea-b78a-875633def6f5\") "
Sep 09 12:39:43.561158 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:43.561135    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadcf98a-5570-4cea-b78a-875633def6f5-kube-api-access-75qvt" (OuterVolumeSpecName: "kube-api-access-75qvt") pod "fadcf98a-5570-4cea-b78a-875633def6f5" (UID: "fadcf98a-5570-4cea-b78a-875633def6f5"). InnerVolumeSpecName "kube-api-access-75qvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:39:43.660447 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:43.660422    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75qvt\" (UniqueName: \"kubernetes.io/projected/fadcf98a-5570-4cea-b78a-875633def6f5-kube-api-access-75qvt\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:39:43.921698 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:43.921600    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7672-d4481a"]
Sep 09 12:39:43.924057 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:43.924028    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7672-d4481a"]
Sep 09 12:39:44.036313 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.036281    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7672-0f320c"]
Sep 09 12:39:44.036752 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.036738    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7724f38b-7333-4043-b67b-188ae27e9841" containerName="test-curl-7672-d4481a"
Sep 09 12:39:44.036791 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.036755    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7724f38b-7333-4043-b67b-188ae27e9841" containerName="test-curl-7672-d4481a"
Sep 09 12:39:44.036834 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.036825    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7724f38b-7333-4043-b67b-188ae27e9841" containerName="test-curl-7672-d4481a"
Sep 09 12:39:44.040021 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.040007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-0f320c"
Sep 09 12:39:44.045940 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.045913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7672-0f320c"]
Sep 09 12:39:44.064480 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.064447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5bmt\" (UniqueName: \"kubernetes.io/projected/0fb91565-fb84-4e0b-ae90-5433b61c597f-kube-api-access-c5bmt\") pod \"test-curl-7672-0f320c\" (UID: \"0fb91565-fb84-4e0b-ae90-5433b61c597f\") " pod="opendatahub/test-curl-7672-0f320c"
Sep 09 12:39:44.165202 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.165170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5bmt\" (UniqueName: \"kubernetes.io/projected/0fb91565-fb84-4e0b-ae90-5433b61c597f-kube-api-access-c5bmt\") pod \"test-curl-7672-0f320c\" (UID: \"0fb91565-fb84-4e0b-ae90-5433b61c597f\") " pod="opendatahub/test-curl-7672-0f320c"
Sep 09 12:39:44.176562 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.176504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5bmt\" (UniqueName: \"kubernetes.io/projected/0fb91565-fb84-4e0b-ae90-5433b61c597f-kube-api-access-c5bmt\") pod \"test-curl-7672-0f320c\" (UID: \"0fb91565-fb84-4e0b-ae90-5433b61c597f\") " pod="opendatahub/test-curl-7672-0f320c"
Sep 09 12:39:44.187096 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.187064    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fadcf98a-5570-4cea-b78a-875633def6f5" path="/var/lib/kubelet/pods/fadcf98a-5570-4cea-b78a-875633def6f5/volumes"
Sep 09 12:39:44.333788 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.333757    2578 generic.go:356] "Generic (PLEG): container finished" podID="fadcf98a-5570-4cea-b78a-875633def6f5" containerID="a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4" exitCode=137
Sep 09 12:39:44.334246 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.333801    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-15555a"
Sep 09 12:39:44.334246 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.333857    2578 scope.go:117] "RemoveContainer" containerID="a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4"
Sep 09 12:39:44.334246 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:39:44.334011    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7672-d4481a" container="test-curl-7672-d4481a"
Sep 09 12:39:44.347050 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.347006    2578 scope.go:117] "RemoveContainer" containerID="a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4"
Sep 09 12:39:44.347513 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:39:44.347484    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4\": container with ID starting with a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4 not found: ID does not exist" containerID="a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4"
Sep 09 12:39:44.347610 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.347526    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4"} err="failed to get container status \"a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4\": rpc error: code = NotFound desc = could not find container \"a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4\": container with ID starting with a7a5acefee6c1c6900846c6cc59f2a871d95c98559799b887a72497edc6e11a4 not found: ID does not exist"
Sep 09 12:39:44.351435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.351403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-0f320c"
Sep 09 12:39:44.625061 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:44.625037    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7672-0f320c"]
Sep 09 12:39:44.627295 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:39:44.627266    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fb91565_fb84_4e0b_ae90_5433b61c597f.slice/crio-f926a61d093b3d3d458d660e46b483ca7c2a261ec66e90f924139c9fd848840a WatchSource:0}: Error finding container f926a61d093b3d3d458d660e46b483ca7c2a261ec66e90f924139c9fd848840a: Status 404 returned error can't find the container with id f926a61d093b3d3d458d660e46b483ca7c2a261ec66e90f924139c9fd848840a
Sep 09 12:39:45.343065 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:45.343033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7672-0f320c" event={"ID":"0fb91565-fb84-4e0b-ae90-5433b61c597f","Type":"ContainerStarted","Data":"be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657"}
Sep 09 12:39:45.343475 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:45.343077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7672-0f320c" event={"ID":"0fb91565-fb84-4e0b-ae90-5433b61c597f","Type":"ContainerStarted","Data":"f926a61d093b3d3d458d660e46b483ca7c2a261ec66e90f924139c9fd848840a"}
Sep 09 12:39:45.343475 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:45.343170    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7672-d4481a" podUID="7724f38b-7333-4043-b67b-188ae27e9841" containerName="test-curl-7672-d4481a" containerID="cri-o://930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22" gracePeriod=2
Sep 09 12:39:45.360292 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:45.360241    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7672-0f320c" podStartSLOduration=1.3602053889999999 podStartE2EDuration="1.360205389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:39:45.358670764 +0000 UTC m=+1205.702268715" watchObservedRunningTime="2026-09-09 12:39:45.360205389 +0000 UTC m=+1205.703803336"
Sep 09 12:39:45.501742 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:45.501717    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-281116"
Sep 09 12:39:45.577137 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:45.577104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zlgq\" (UniqueName: \"kubernetes.io/projected/a4694532-34f0-478f-a92a-9e0f8c6415b4-kube-api-access-6zlgq\") pod \"a4694532-34f0-478f-a92a-9e0f8c6415b4\" (UID: \"a4694532-34f0-478f-a92a-9e0f8c6415b4\") "
Sep 09 12:39:45.579207 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:45.579179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4694532-34f0-478f-a92a-9e0f8c6415b4-kube-api-access-6zlgq" (OuterVolumeSpecName: "kube-api-access-6zlgq") pod "a4694532-34f0-478f-a92a-9e0f8c6415b4" (UID: "a4694532-34f0-478f-a92a-9e0f8c6415b4"). InnerVolumeSpecName "kube-api-access-6zlgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:39:45.678891 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:45.678859    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zlgq\" (UniqueName: \"kubernetes.io/projected/a4694532-34f0-478f-a92a-9e0f8c6415b4-kube-api-access-6zlgq\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:39:45.954364 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:45.954332    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7672-0f320c"]
Sep 09 12:39:45.958969 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:45.958944    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7672-0f320c"]
Sep 09 12:39:46.186438 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:46.186406    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4694532-34f0-478f-a92a-9e0f8c6415b4" path="/var/lib/kubelet/pods/a4694532-34f0-478f-a92a-9e0f8c6415b4/volumes"
Sep 09 12:39:46.349531 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:46.349441    2578 generic.go:356] "Generic (PLEG): container finished" podID="a4694532-34f0-478f-a92a-9e0f8c6415b4" containerID="4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db" exitCode=137
Sep 09 12:39:46.349531 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:46.349491    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-281116"
Sep 09 12:39:46.350007 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:46.349550    2578 scope.go:117] "RemoveContainer" containerID="4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db"
Sep 09 12:39:46.350007 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:39:46.349715    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7672-0f320c" container="test-curl-7672-0f320c"
Sep 09 12:39:46.358456 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:46.358439    2578 scope.go:117] "RemoveContainer" containerID="4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db"
Sep 09 12:39:46.358728 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:39:46.358707    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db\": container with ID starting with 4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db not found: ID does not exist" containerID="4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db"
Sep 09 12:39:46.358790 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:46.358738    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db"} err="failed to get container status \"4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db\": rpc error: code = NotFound desc = could not find container \"4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db\": container with ID starting with 4b948c2a8645e6f2197ef066015fe401e6adeaa8963813540f34ab685b56b0db not found: ID does not exist"
Sep 09 12:39:47.357997 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:47.357935    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7672-0f320c" podUID="0fb91565-fb84-4e0b-ae90-5433b61c597f" containerName="test-curl-7672-0f320c" containerID="cri-o://be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657" gracePeriod=2
Sep 09 12:39:47.492651 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:47.492627    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-d4481a"
Sep 09 12:39:47.596292 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:47.596255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfb27\" (UniqueName: \"kubernetes.io/projected/7724f38b-7333-4043-b67b-188ae27e9841-kube-api-access-wfb27\") pod \"7724f38b-7333-4043-b67b-188ae27e9841\" (UID: \"7724f38b-7333-4043-b67b-188ae27e9841\") "
Sep 09 12:39:47.598467 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:47.598399    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7724f38b-7333-4043-b67b-188ae27e9841-kube-api-access-wfb27" (OuterVolumeSpecName: "kube-api-access-wfb27") pod "7724f38b-7333-4043-b67b-188ae27e9841" (UID: "7724f38b-7333-4043-b67b-188ae27e9841"). InnerVolumeSpecName "kube-api-access-wfb27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:39:47.697079 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:47.697042    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfb27\" (UniqueName: \"kubernetes.io/projected/7724f38b-7333-4043-b67b-188ae27e9841-kube-api-access-wfb27\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:39:48.185816 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:48.185783    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7724f38b-7333-4043-b67b-188ae27e9841" path="/var/lib/kubelet/pods/7724f38b-7333-4043-b67b-188ae27e9841/volumes"
Sep 09 12:39:48.365991 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:48.365957    2578 generic.go:356] "Generic (PLEG): container finished" podID="7724f38b-7333-4043-b67b-188ae27e9841" containerID="930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22" exitCode=137
Sep 09 12:39:48.366485 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:48.365999    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-d4481a"
Sep 09 12:39:48.366485 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:48.366062    2578 scope.go:117] "RemoveContainer" containerID="930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22"
Sep 09 12:39:48.375204 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:48.375188    2578 scope.go:117] "RemoveContainer" containerID="930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22"
Sep 09 12:39:48.375606 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:39:48.375587    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22\": container with ID starting with 930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22 not found: ID does not exist" containerID="930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22"
Sep 09 12:39:48.375693 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:48.375616    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22"} err="failed to get container status \"930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22\": rpc error: code = NotFound desc = could not find container \"930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22\": container with ID starting with 930b79696eb7549919a08eb1501b5a7da2c66fa418353132ac02c69c190a8b22 not found: ID does not exist"
Sep 09 12:39:49.550295 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:49.550272    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-0f320c"
Sep 09 12:39:49.615143 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:49.615041    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5bmt\" (UniqueName: \"kubernetes.io/projected/0fb91565-fb84-4e0b-ae90-5433b61c597f-kube-api-access-c5bmt\") pod \"0fb91565-fb84-4e0b-ae90-5433b61c597f\" (UID: \"0fb91565-fb84-4e0b-ae90-5433b61c597f\") "
Sep 09 12:39:49.617268 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:49.617247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb91565-fb84-4e0b-ae90-5433b61c597f-kube-api-access-c5bmt" (OuterVolumeSpecName: "kube-api-access-c5bmt") pod "0fb91565-fb84-4e0b-ae90-5433b61c597f" (UID: "0fb91565-fb84-4e0b-ae90-5433b61c597f"). InnerVolumeSpecName "kube-api-access-c5bmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:39:49.716915 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:49.716886    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5bmt\" (UniqueName: \"kubernetes.io/projected/0fb91565-fb84-4e0b-ae90-5433b61c597f-kube-api-access-c5bmt\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:39:50.192906 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.192871    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fb91565-fb84-4e0b-ae90-5433b61c597f" path="/var/lib/kubelet/pods/0fb91565-fb84-4e0b-ae90-5433b61c597f/volumes"
Sep 09 12:39:50.381695 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.381655    2578 generic.go:356] "Generic (PLEG): container finished" podID="0fb91565-fb84-4e0b-ae90-5433b61c597f" containerID="be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657" exitCode=137
Sep 09 12:39:50.381695 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.381697    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7672-0f320c"
Sep 09 12:39:50.381947 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.381739    2578 scope.go:117] "RemoveContainer" containerID="be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657"
Sep 09 12:39:50.390250 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.390232    2578 scope.go:117] "RemoveContainer" containerID="be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657"
Sep 09 12:39:50.390550 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:39:50.390529    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657\": container with ID starting with be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657 not found: ID does not exist" containerID="be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657"
Sep 09 12:39:50.390608 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.390559    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657"} err="failed to get container status \"be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657\": rpc error: code = NotFound desc = could not find container \"be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657\": container with ID starting with be326d3c66910327ad500a3f1ae9a15bcc485a3d6f66c6ef72625c62b6a1c657 not found: ID does not exist"
Sep 09 12:39:50.841121 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.841087    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"]
Sep 09 12:39:50.841762 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.841745    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb91565-fb84-4e0b-ae90-5433b61c597f" containerName="test-curl-7672-0f320c"
Sep 09 12:39:50.841808 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.841764    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb91565-fb84-4e0b-ae90-5433b61c597f" containerName="test-curl-7672-0f320c"
Sep 09 12:39:50.841901 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.841890    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fb91565-fb84-4e0b-ae90-5433b61c597f" containerName="test-curl-7672-0f320c"
Sep 09 12:39:50.846746 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.846721    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:50.848661 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.848642    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6a4af97c-serving-cert\""
Sep 09 12:39:50.853739 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.853715    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"]
Sep 09 12:39:50.928196 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.928158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9de8f22-4c45-420a-8c32-a7f2043d8588-openshift-service-ca\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:50.928361 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.928246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-metrics-certs\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:50.928361 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.928270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-maas-api-tls\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:50.928361 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:50.928284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd47g\" (UniqueName: \"kubernetes.io/projected/e9de8f22-4c45-420a-8c32-a7f2043d8588-kube-api-access-sd47g\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:51.029703 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.029666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9de8f22-4c45-420a-8c32-a7f2043d8588-openshift-service-ca\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:51.029861 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.029739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-metrics-certs\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:51.029861 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.029769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-maas-api-tls\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:51.029861 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.029793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd47g\" (UniqueName: \"kubernetes.io/projected/e9de8f22-4c45-420a-8c32-a7f2043d8588-kube-api-access-sd47g\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:51.030503 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.030481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9de8f22-4c45-420a-8c32-a7f2043d8588-openshift-service-ca\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:51.032406 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.032372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-metrics-certs\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:51.032545 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.032527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-maas-api-tls\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:51.039931 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.039906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd47g\" (UniqueName: \"kubernetes.io/projected/e9de8f22-4c45-420a-8c32-a7f2043d8588-kube-api-access-sd47g\") pod \"maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:51.159634 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.159604    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:51.429813 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.429791    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"]
Sep 09 12:39:51.432060 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:39:51.432027    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9de8f22_4c45_420a_8c32_a7f2043d8588.slice/crio-903507ffcd891c910720423a901e3055ee7054d3ce24ed9a832648db1c9a3a34 WatchSource:0}: Error finding container 903507ffcd891c910720423a901e3055ee7054d3ce24ed9a832648db1c9a3a34: Status 404 returned error can't find the container with id 903507ffcd891c910720423a901e3055ee7054d3ce24ed9a832648db1c9a3a34
Sep 09 12:39:51.511150 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.511110    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"]
Sep 09 12:39:51.515912 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.515896    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:39:51.518034 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.518004    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6a4af97c\""
Sep 09 12:39:51.518173 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.518044    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6a4af97c-dockercfg-54j45\""
Sep 09 12:39:51.527730 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.527708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"]
Sep 09 12:39:51.635324 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.635290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3d00f6-ed9a-46ff-b5fc-619352877a03-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx\" (UID: \"2c3d00f6-ed9a-46ff-b5fc-619352877a03\") " pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:39:51.635512 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.635363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwlbg\" (UniqueName: \"kubernetes.io/projected/2c3d00f6-ed9a-46ff-b5fc-619352877a03-kube-api-access-kwlbg\") pod \"payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx\" (UID: \"2c3d00f6-ed9a-46ff-b5fc-619352877a03\") " pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:39:51.735998 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.735914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwlbg\" (UniqueName: \"kubernetes.io/projected/2c3d00f6-ed9a-46ff-b5fc-619352877a03-kube-api-access-kwlbg\") pod \"payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx\" (UID: \"2c3d00f6-ed9a-46ff-b5fc-619352877a03\") " pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:39:51.736149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.736003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3d00f6-ed9a-46ff-b5fc-619352877a03-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx\" (UID: \"2c3d00f6-ed9a-46ff-b5fc-619352877a03\") " pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:39:51.736703 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.736683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3d00f6-ed9a-46ff-b5fc-619352877a03-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx\" (UID: \"2c3d00f6-ed9a-46ff-b5fc-619352877a03\") " pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:39:51.746324 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.746299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwlbg\" (UniqueName: \"kubernetes.io/projected/2c3d00f6-ed9a-46ff-b5fc-619352877a03-kube-api-access-kwlbg\") pod \"payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx\" (UID: \"2c3d00f6-ed9a-46ff-b5fc-619352877a03\") " pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:39:51.826631 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:51.826600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:39:52.097207 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:52.097086    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"]
Sep 09 12:39:52.099379 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:39:52.099355    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3d00f6_ed9a_46ff_b5fc_619352877a03.slice/crio-51e9e42b8272412ac750dda9179d8694da6d1bdd3d6a493e11e0c1eff46b944a WatchSource:0}: Error finding container 51e9e42b8272412ac750dda9179d8694da6d1bdd3d6a493e11e0c1eff46b944a: Status 404 returned error can't find the container with id 51e9e42b8272412ac750dda9179d8694da6d1bdd3d6a493e11e0c1eff46b944a
Sep 09 12:39:52.395471 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:52.395373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx" event={"ID":"2c3d00f6-ed9a-46ff-b5fc-619352877a03","Type":"ContainerStarted","Data":"149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021"}
Sep 09 12:39:52.395471 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:52.395428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx" event={"ID":"2c3d00f6-ed9a-46ff-b5fc-619352877a03","Type":"ContainerStarted","Data":"51e9e42b8272412ac750dda9179d8694da6d1bdd3d6a493e11e0c1eff46b944a"}
Sep 09 12:39:52.395689 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:52.395487    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:39:52.397240 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:52.397188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z" event={"ID":"e9de8f22-4c45-420a-8c32-a7f2043d8588","Type":"ContainerStarted","Data":"2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58"}
Sep 09 12:39:52.397360 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:52.397260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z" event={"ID":"e9de8f22-4c45-420a-8c32-a7f2043d8588","Type":"ContainerStarted","Data":"903507ffcd891c910720423a901e3055ee7054d3ce24ed9a832648db1c9a3a34"}
Sep 09 12:39:52.397360 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:52.397293    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:39:52.412988 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:52.412928    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx" podStartSLOduration=1.412911771 podStartE2EDuration="1.412911771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:39:52.410666898 +0000 UTC m=+1212.754264845" watchObservedRunningTime="2026-09-09 12:39:52.412911771 +0000 UTC m=+1212.756509719"
Sep 09 12:39:52.427174 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:52.427124    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z" podStartSLOduration=1.734987103 podStartE2EDuration="2.427107276s" totalImagesPullingTime="692.120173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:39:52.426117241 +0000 UTC m=+1212.769715204" watchObservedRunningTime="2026-09-09 12:39:52.427107276 +0000 UTC m=+1212.770705243"
Sep 09 12:39:54.932437 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:54.932402    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:39:58.408107 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:39:58.408079    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:40:03.403983 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:03.403953    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:40:28.254427 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.254385    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"]
Sep 09 12:40:28.258642 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.258616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.261472 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.261449    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fe6606\"/\"kube-root-ca.crt\""
Sep 09 12:40:28.261613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.261499    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fe6606\"/\"auth-test-model-fe6606-kserve-self-signed-certs\""
Sep 09 12:40:28.261701 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.261679    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fe6606\"/\"default-dockercfg-vxvmq\""
Sep 09 12:40:28.261776 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.261757    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fe6606\"/\"openshift-service-ca.crt\""
Sep 09 12:40:28.272974 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.272947    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"]
Sep 09 12:40:28.296602 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.296571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-dshm\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.296752 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.296617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tmp-dir\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.296752 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.296680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-model-cache\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.296752 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.296718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnql\" (UniqueName: \"kubernetes.io/projected/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-kube-api-access-xfnql\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.296931 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.296828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tls-certs\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.296985 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.296939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-home\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.397988 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.397955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tls-certs\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.398149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.398032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-home\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.398149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.398074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-dshm\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.398149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.398092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tmp-dir\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.398149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.398110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-model-cache\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.398149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.398142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnql\" (UniqueName: \"kubernetes.io/projected/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-kube-api-access-xfnql\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.398555 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.398535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-home\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.398687 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.398607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-model-cache\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.398687 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.398651    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tmp-dir\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.400572 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.400548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-dshm\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.400708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.400689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tls-certs\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.406860 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.406840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnql\" (UniqueName: \"kubernetes.io/projected/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-kube-api-access-xfnql\") pod \"auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.571130 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.571042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:28.846531 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:28.846501    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"]
Sep 09 12:40:28.847964 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:40:28.847937    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod285dc3dd_5faf_480b_9629_a6f7dc0c4b24.slice/crio-4d6e536fa01e36ff347c3abaf61842314e867999d514a736fa45f53dd1ff9b2e WatchSource:0}: Error finding container 4d6e536fa01e36ff347c3abaf61842314e867999d514a736fa45f53dd1ff9b2e: Status 404 returned error can't find the container with id 4d6e536fa01e36ff347c3abaf61842314e867999d514a736fa45f53dd1ff9b2e
Sep 09 12:40:29.596951 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:29.596914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4" event={"ID":"285dc3dd-5faf-480b-9629-a6f7dc0c4b24","Type":"ContainerStarted","Data":"5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb"}
Sep 09 12:40:29.597383 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:29.596961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4" event={"ID":"285dc3dd-5faf-480b-9629-a6f7dc0c4b24","Type":"ContainerStarted","Data":"4d6e536fa01e36ff347c3abaf61842314e867999d514a736fa45f53dd1ff9b2e"}
Sep 09 12:40:29.615301 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:29.615253    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4" podStartSLOduration=1.6152373199999999 podStartE2EDuration="1.61523732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:40:29.613049362 +0000 UTC m=+1249.956647307" watchObservedRunningTime="2026-09-09 12:40:29.61523732 +0000 UTC m=+1249.958835267"
Sep 09 12:40:33.225502 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:33.225425    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:40:38.571921 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:38.571886    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:38.572444 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:38.571941    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:38.585092 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:38.585069    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:38.656953 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:38.656925    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:40:55.738095 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:40:55.738066    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:41:16.732209 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:16.732172    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:41:31.528151 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:31.528098    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:41:40.533431 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.533397    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"]
Sep 09 12:41:40.533915 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.533655    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z" podUID="e9de8f22-4c45-420a-8c32-a7f2043d8588" containerName="maas-api" containerID="cri-o://2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58" gracePeriod=30
Sep 09 12:41:40.588270 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.588240    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"]
Sep 09 12:41:40.588486 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.588450    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx" podUID="2c3d00f6-ed9a-46ff-b5fc-619352877a03" containerName="payload-processing" containerID="cri-o://149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021" gracePeriod=30
Sep 09 12:41:40.854465 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.854443    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:41:40.857615 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.857595    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:41:40.920567 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.920533    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwlbg\" (UniqueName: \"kubernetes.io/projected/2c3d00f6-ed9a-46ff-b5fc-619352877a03-kube-api-access-kwlbg\") pod \"2c3d00f6-ed9a-46ff-b5fc-619352877a03\" (UID: \"2c3d00f6-ed9a-46ff-b5fc-619352877a03\") "
Sep 09 12:41:40.920567 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.920570    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-metrics-certs\") pod \"e9de8f22-4c45-420a-8c32-a7f2043d8588\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") "
Sep 09 12:41:40.920821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.920619    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-maas-api-tls\") pod \"e9de8f22-4c45-420a-8c32-a7f2043d8588\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") "
Sep 09 12:41:40.920821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.920683    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3d00f6-ed9a-46ff-b5fc-619352877a03-plugins-config-volume\") pod \"2c3d00f6-ed9a-46ff-b5fc-619352877a03\" (UID: \"2c3d00f6-ed9a-46ff-b5fc-619352877a03\") "
Sep 09 12:41:40.920821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.920766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd47g\" (UniqueName: \"kubernetes.io/projected/e9de8f22-4c45-420a-8c32-a7f2043d8588-kube-api-access-sd47g\") pod \"e9de8f22-4c45-420a-8c32-a7f2043d8588\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") "
Sep 09 12:41:40.920821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.920811    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9de8f22-4c45-420a-8c32-a7f2043d8588-openshift-service-ca\") pod \"e9de8f22-4c45-420a-8c32-a7f2043d8588\" (UID: \"e9de8f22-4c45-420a-8c32-a7f2043d8588\") "
Sep 09 12:41:40.921152 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.921122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3d00f6-ed9a-46ff-b5fc-619352877a03-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c3d00f6-ed9a-46ff-b5fc-619352877a03" (UID: "2c3d00f6-ed9a-46ff-b5fc-619352877a03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:41:40.921658 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.921627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9de8f22-4c45-420a-8c32-a7f2043d8588-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e9de8f22-4c45-420a-8c32-a7f2043d8588" (UID: "e9de8f22-4c45-420a-8c32-a7f2043d8588"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:41:40.923007 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.922972    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3d00f6-ed9a-46ff-b5fc-619352877a03-kube-api-access-kwlbg" (OuterVolumeSpecName: "kube-api-access-kwlbg") pod "2c3d00f6-ed9a-46ff-b5fc-619352877a03" (UID: "2c3d00f6-ed9a-46ff-b5fc-619352877a03"). InnerVolumeSpecName "kube-api-access-kwlbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:41:40.923092 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.922980    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9de8f22-4c45-420a-8c32-a7f2043d8588" (UID: "e9de8f22-4c45-420a-8c32-a7f2043d8588"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:41:40.923504 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.923484    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e9de8f22-4c45-420a-8c32-a7f2043d8588" (UID: "e9de8f22-4c45-420a-8c32-a7f2043d8588"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:41:40.923569 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.923503    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9de8f22-4c45-420a-8c32-a7f2043d8588-kube-api-access-sd47g" (OuterVolumeSpecName: "kube-api-access-sd47g") pod "e9de8f22-4c45-420a-8c32-a7f2043d8588" (UID: "e9de8f22-4c45-420a-8c32-a7f2043d8588"). InnerVolumeSpecName "kube-api-access-sd47g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:41:40.972040 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.972017    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c3d00f6-ed9a-46ff-b5fc-619352877a03" containerID="149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021" exitCode=0
Sep 09 12:41:40.972167 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.972070    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"
Sep 09 12:41:40.972167 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.972094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx" event={"ID":"2c3d00f6-ed9a-46ff-b5fc-619352877a03","Type":"ContainerDied","Data":"149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021"}
Sep 09 12:41:40.972316 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.972167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx" event={"ID":"2c3d00f6-ed9a-46ff-b5fc-619352877a03","Type":"ContainerDied","Data":"51e9e42b8272412ac750dda9179d8694da6d1bdd3d6a493e11e0c1eff46b944a"}
Sep 09 12:41:40.972316 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.972193    2578 scope.go:117] "RemoveContainer" containerID="149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021"
Sep 09 12:41:40.973468 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.973449    2578 generic.go:356] "Generic (PLEG): container finished" podID="e9de8f22-4c45-420a-8c32-a7f2043d8588" containerID="2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58" exitCode=0
Sep 09 12:41:40.973569 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.973517    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"
Sep 09 12:41:40.973569 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.973545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z" event={"ID":"e9de8f22-4c45-420a-8c32-a7f2043d8588","Type":"ContainerDied","Data":"2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58"}
Sep 09 12:41:40.973844 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.973591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z" event={"ID":"e9de8f22-4c45-420a-8c32-a7f2043d8588","Type":"ContainerDied","Data":"903507ffcd891c910720423a901e3055ee7054d3ce24ed9a832648db1c9a3a34"}
Sep 09 12:41:40.981285 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.981267    2578 scope.go:117] "RemoveContainer" containerID="149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021"
Sep 09 12:41:40.981589 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:41:40.981569    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021\": container with ID starting with 149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021 not found: ID does not exist" containerID="149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021"
Sep 09 12:41:40.981645 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.981596    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021"} err="failed to get container status \"149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021\": rpc error: code = NotFound desc = could not find container \"149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021\": container with ID starting with 149cc50118b281d80087c1f4286e471eeb51d3f22413834f2ff4eaf87f27c021 not found: ID does not exist"
Sep 09 12:41:40.981645 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.981612    2578 scope.go:117] "RemoveContainer" containerID="2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58"
Sep 09 12:41:40.990341 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.990324    2578 scope.go:117] "RemoveContainer" containerID="2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58"
Sep 09 12:41:40.990601 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:41:40.990582    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58\": container with ID starting with 2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58 not found: ID does not exist" containerID="2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58"
Sep 09 12:41:40.990674 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:40.990609    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58"} err="failed to get container status \"2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58\": rpc error: code = NotFound desc = could not find container \"2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58\": container with ID starting with 2bbbb0f595e655f3fe365f9cbac46b9127aae9b88ea95cfa447e4bdbf4509b58 not found: ID does not exist"
Sep 09 12:41:41.002169 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:41.002150    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"]
Sep 09 12:41:41.005777 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:41.005758    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a4af97c-85f89b5f69-4jlvx"]
Sep 09 12:41:41.016184 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:41.016164    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"]
Sep 09 12:41:41.020027 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:41.020005    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a4af97c-658db9b654-ntc8z"]
Sep 09 12:41:41.022427 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:41.022399    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd47g\" (UniqueName: \"kubernetes.io/projected/e9de8f22-4c45-420a-8c32-a7f2043d8588-kube-api-access-sd47g\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:41:41.022427 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:41.022431    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9de8f22-4c45-420a-8c32-a7f2043d8588-openshift-service-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:41:41.022526 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:41.022442    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwlbg\" (UniqueName: \"kubernetes.io/projected/2c3d00f6-ed9a-46ff-b5fc-619352877a03-kube-api-access-kwlbg\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:41:41.022526 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:41.022459    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-metrics-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:41:41.022526 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:41.022467    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9de8f22-4c45-420a-8c32-a7f2043d8588-maas-api-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:41:41.022526 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:41.022475    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3d00f6-ed9a-46ff-b5fc-619352877a03-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:41:42.186397 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:42.186367    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c3d00f6-ed9a-46ff-b5fc-619352877a03" path="/var/lib/kubelet/pods/2c3d00f6-ed9a-46ff-b5fc-619352877a03/volumes"
Sep 09 12:41:42.186770 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:42.186759    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9de8f22-4c45-420a-8c32-a7f2043d8588" path="/var/lib/kubelet/pods/e9de8f22-4c45-420a-8c32-a7f2043d8588/volumes"
Sep 09 12:41:49.224745 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:41:49.224715    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:42:05.226559 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:05.226480    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:42:20.824415 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:20.824381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:42:39.826691 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:39.826658    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:42:49.556100 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.556058    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7675-099c59"]
Sep 09 12:42:49.556691 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.556670    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9de8f22-4c45-420a-8c32-a7f2043d8588" containerName="maas-api"
Sep 09 12:42:49.556790 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.556694    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9de8f22-4c45-420a-8c32-a7f2043d8588" containerName="maas-api"
Sep 09 12:42:49.556790 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.556722    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3d00f6-ed9a-46ff-b5fc-619352877a03" containerName="payload-processing"
Sep 09 12:42:49.556790 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.556731    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3d00f6-ed9a-46ff-b5fc-619352877a03" containerName="payload-processing"
Sep 09 12:42:49.556940 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.556854    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9de8f22-4c45-420a-8c32-a7f2043d8588" containerName="maas-api"
Sep 09 12:42:49.556940 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.556875    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c3d00f6-ed9a-46ff-b5fc-619352877a03" containerName="payload-processing"
Sep 09 12:42:49.561009 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.560987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-099c59"
Sep 09 12:42:49.563186 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.563161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4dspl\""
Sep 09 12:42:49.565801 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.565778    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7675-099c59"]
Sep 09 12:42:49.590739 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.590711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rjn\" (UniqueName: \"kubernetes.io/projected/eff30006-8aa0-4012-b150-0ccd7d080742-kube-api-access-j8rjn\") pod \"test-curl-7675-099c59\" (UID: \"eff30006-8aa0-4012-b150-0ccd7d080742\") " pod="opendatahub/test-curl-7675-099c59"
Sep 09 12:42:49.692203 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.692168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rjn\" (UniqueName: \"kubernetes.io/projected/eff30006-8aa0-4012-b150-0ccd7d080742-kube-api-access-j8rjn\") pod \"test-curl-7675-099c59\" (UID: \"eff30006-8aa0-4012-b150-0ccd7d080742\") " pod="opendatahub/test-curl-7675-099c59"
Sep 09 12:42:49.700063 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.700039    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rjn\" (UniqueName: \"kubernetes.io/projected/eff30006-8aa0-4012-b150-0ccd7d080742-kube-api-access-j8rjn\") pod \"test-curl-7675-099c59\" (UID: \"eff30006-8aa0-4012-b150-0ccd7d080742\") " pod="opendatahub/test-curl-7675-099c59"
Sep 09 12:42:49.871549 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:49.871460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-099c59"
Sep 09 12:42:50.143262 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:50.143205    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7675-099c59"]
Sep 09 12:42:50.145560 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:42:50.145529    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeff30006_8aa0_4012_b150_0ccd7d080742.slice/crio-c1c0f5ee9cec6f09767e8fbc9221daba3ab96de5e2b52210d6974f5d1a1b9b7f WatchSource:0}: Error finding container c1c0f5ee9cec6f09767e8fbc9221daba3ab96de5e2b52210d6974f5d1a1b9b7f: Status 404 returned error can't find the container with id c1c0f5ee9cec6f09767e8fbc9221daba3ab96de5e2b52210d6974f5d1a1b9b7f
Sep 09 12:42:50.334722 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:50.334679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7675-099c59" event={"ID":"eff30006-8aa0-4012-b150-0ccd7d080742","Type":"ContainerStarted","Data":"7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a"}
Sep 09 12:42:50.334722 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:50.334725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7675-099c59" event={"ID":"eff30006-8aa0-4012-b150-0ccd7d080742","Type":"ContainerStarted","Data":"c1c0f5ee9cec6f09767e8fbc9221daba3ab96de5e2b52210d6974f5d1a1b9b7f"}
Sep 09 12:42:50.351064 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:50.351003    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7675-099c59" podStartSLOduration=1.35098737 podStartE2EDuration="1.35098737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:42:50.348742416 +0000 UTC m=+1390.692340364" watchObservedRunningTime="2026-09-09 12:42:50.35098737 +0000 UTC m=+1390.694585312"
Sep 09 12:42:50.944786 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:50.944735    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7675-099c59"]
Sep 09 12:42:50.949970 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:50.949941    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7675-099c59"]
Sep 09 12:42:51.339862 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:42:51.339781    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7675-099c59" container="test-curl-7675-099c59"
Sep 09 12:42:52.344284 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:52.344204    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7675-099c59" podUID="eff30006-8aa0-4012-b150-0ccd7d080742" containerName="test-curl-7675-099c59" containerID="cri-o://7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a" gracePeriod=2
Sep 09 12:42:53.446791 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:53.446716    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"]
Sep 09 12:42:54.534242 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:54.534194    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-099c59"
Sep 09 12:42:54.660941 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:54.660899    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8rjn\" (UniqueName: \"kubernetes.io/projected/eff30006-8aa0-4012-b150-0ccd7d080742-kube-api-access-j8rjn\") pod \"eff30006-8aa0-4012-b150-0ccd7d080742\" (UID: \"eff30006-8aa0-4012-b150-0ccd7d080742\") "
Sep 09 12:42:54.663149 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:54.663123    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff30006-8aa0-4012-b150-0ccd7d080742-kube-api-access-j8rjn" (OuterVolumeSpecName: "kube-api-access-j8rjn") pod "eff30006-8aa0-4012-b150-0ccd7d080742" (UID: "eff30006-8aa0-4012-b150-0ccd7d080742"). InnerVolumeSpecName "kube-api-access-j8rjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:42:54.762099 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:54.762055    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rjn\" (UniqueName: \"kubernetes.io/projected/eff30006-8aa0-4012-b150-0ccd7d080742-kube-api-access-j8rjn\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:42:55.025502 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:55.025417    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:42:55.361545 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:55.361464    2578 generic.go:356] "Generic (PLEG): container finished" podID="eff30006-8aa0-4012-b150-0ccd7d080742" containerID="7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a" exitCode=137
Sep 09 12:42:55.361545 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:55.361506    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-099c59"
Sep 09 12:42:55.361545 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:55.361538    2578 scope.go:117] "RemoveContainer" containerID="7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a"
Sep 09 12:42:55.370778 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:55.370760    2578 scope.go:117] "RemoveContainer" containerID="7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a"
Sep 09 12:42:55.371024 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:42:55.371007    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a\": container with ID starting with 7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a not found: ID does not exist" containerID="7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a"
Sep 09 12:42:55.371076 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:55.371032    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a"} err="failed to get container status \"7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a\": rpc error: code = NotFound desc = could not find container \"7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a\": container with ID starting with 7141399d652a52968f19e4942fc83b4db42447fb15fa83df98a59391f3050d7a not found: ID does not exist"
Sep 09 12:42:55.371320 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:42:55.371296    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7675-099c59\" is forbidden: User \"system:node:ip-10-0-133-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-152.ec2.internal' and this object" podUID="eff30006-8aa0-4012-b150-0ccd7d080742" pod="opendatahub/test-curl-7675-099c59"
Sep 09 12:42:56.186451 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:42:56.186422    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff30006-8aa0-4012-b150-0ccd7d080742" path="/var/lib/kubelet/pods/eff30006-8aa0-4012-b150-0ccd7d080742/volumes"
Sep 09 12:43:05.436371 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.436338    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"]
Sep 09 12:43:05.436784 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.436773    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff30006-8aa0-4012-b150-0ccd7d080742" containerName="test-curl-7675-099c59"
Sep 09 12:43:05.436784 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.436784    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff30006-8aa0-4012-b150-0ccd7d080742" containerName="test-curl-7675-099c59"
Sep 09 12:43:05.436873 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.436851    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff30006-8aa0-4012-b150-0ccd7d080742" containerName="test-curl-7675-099c59"
Sep 09 12:43:05.441574 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.441558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.443738 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.443712    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-283adb\"/\"openshift-service-ca.crt\""
Sep 09 12:43:05.443879 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.443793    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-283adb\"/\"kube-root-ca.crt\""
Sep 09 12:43:05.444329 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.444309    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-283adb\"/\"sub-test-model-283adb-kserve-self-signed-certs\""
Sep 09 12:43:05.444329 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.444320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-283adb\"/\"default-dockercfg-8bz29\""
Sep 09 12:43:05.449585 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.449561    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"]
Sep 09 12:43:05.576134 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.576091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tls-certs\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.576134 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.576133    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-home\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.576387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.576155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tmp-dir\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.576387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.576207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-dshm\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.576387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.576313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmtnb\" (UniqueName: \"kubernetes.io/projected/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-kube-api-access-mmtnb\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.576490 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.576436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-model-cache\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.677551 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.677498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-model-cache\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.677551 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.677550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tls-certs\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.677786 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.677660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-home\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.677786 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.677688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tmp-dir\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.677786 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.677722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-dshm\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.677786 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.677756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmtnb\" (UniqueName: \"kubernetes.io/projected/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-kube-api-access-mmtnb\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.677963 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.677940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-model-cache\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.678011 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.677997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-home\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.678081 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.678065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tmp-dir\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.680117 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.680092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-dshm\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.680351 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.680334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tls-certs\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.685591 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.685570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmtnb\" (UniqueName: \"kubernetes.io/projected/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-kube-api-access-mmtnb\") pod \"sub-test-model-283adb-kserve-67477d6c87-qxsq9\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") " pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:05.752273 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:05.752147    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:06.028322 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:06.028299    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"]
Sep 09 12:43:06.030321 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:43:06.030297    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba3c2c1a_bce3_4e30_9b71_7ad5f8b2303c.slice/crio-3fcbd73edd2915e6b71b159ea2a69e3a3fecf0e66c82b35520c04a01d12145f1 WatchSource:0}: Error finding container 3fcbd73edd2915e6b71b159ea2a69e3a3fecf0e66c82b35520c04a01d12145f1: Status 404 returned error can't find the container with id 3fcbd73edd2915e6b71b159ea2a69e3a3fecf0e66c82b35520c04a01d12145f1
Sep 09 12:43:06.424198 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:06.424161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9" event={"ID":"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c","Type":"ContainerStarted","Data":"e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d"}
Sep 09 12:43:06.424404 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:06.424210    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9" event={"ID":"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c","Type":"ContainerStarted","Data":"3fcbd73edd2915e6b71b159ea2a69e3a3fecf0e66c82b35520c04a01d12145f1"}
Sep 09 12:43:06.445896 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:06.445844    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9" podStartSLOduration=1.445829072 podStartE2EDuration="1.445829072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:43:06.443769051 +0000 UTC m=+1406.787367001" watchObservedRunningTime="2026-09-09 12:43:06.445829072 +0000 UTC m=+1406.789427020"
Sep 09 12:43:08.468781 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.468713    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4" podUID="285dc3dd-5faf-480b-9629-a6f7dc0c4b24" containerName="main" containerID="cri-o://5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb" gracePeriod=45
Sep 09 12:43:08.535467 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.535433    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:43:08.723434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.723376    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:43:08.808203 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.808151    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-model-cache\") pod \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") "
Sep 09 12:43:08.808433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.808271    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfnql\" (UniqueName: \"kubernetes.io/projected/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-kube-api-access-xfnql\") pod \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") "
Sep 09 12:43:08.808433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.808314    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tmp-dir\") pod \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") "
Sep 09 12:43:08.808433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.808353    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-dshm\") pod \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") "
Sep 09 12:43:08.808433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.808404    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tls-certs\") pod \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") "
Sep 09 12:43:08.808433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.808426    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-home\") pod \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\" (UID: \"285dc3dd-5faf-480b-9629-a6f7dc0c4b24\") "
Sep 09 12:43:08.808693 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.808514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-model-cache" (OuterVolumeSpecName: "model-cache") pod "285dc3dd-5faf-480b-9629-a6f7dc0c4b24" (UID: "285dc3dd-5faf-480b-9629-a6f7dc0c4b24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:43:08.808693 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.808606    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "285dc3dd-5faf-480b-9629-a6f7dc0c4b24" (UID: "285dc3dd-5faf-480b-9629-a6f7dc0c4b24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:43:08.808794 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.808760    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-home" (OuterVolumeSpecName: "home") pod "285dc3dd-5faf-480b-9629-a6f7dc0c4b24" (UID: "285dc3dd-5faf-480b-9629-a6f7dc0c4b24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:43:08.808794 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.808788    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-model-cache\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:43:08.808886 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.808807    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tmp-dir\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:43:08.810783 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.810754    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-dshm" (OuterVolumeSpecName: "dshm") pod "285dc3dd-5faf-480b-9629-a6f7dc0c4b24" (UID: "285dc3dd-5faf-480b-9629-a6f7dc0c4b24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:43:08.810783 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.810770    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-kube-api-access-xfnql" (OuterVolumeSpecName: "kube-api-access-xfnql") pod "285dc3dd-5faf-480b-9629-a6f7dc0c4b24" (UID: "285dc3dd-5faf-480b-9629-a6f7dc0c4b24"). InnerVolumeSpecName "kube-api-access-xfnql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:43:08.810903 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.810803    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "285dc3dd-5faf-480b-9629-a6f7dc0c4b24" (UID: "285dc3dd-5faf-480b-9629-a6f7dc0c4b24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:43:08.909657 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.909610    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-dshm\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:43:08.909657 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.909650    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-tls-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:43:08.909657 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.909661    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-home\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:43:08.909657 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:08.909669    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfnql\" (UniqueName: \"kubernetes.io/projected/285dc3dd-5faf-480b-9629-a6f7dc0c4b24-kube-api-access-xfnql\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:43:09.442905 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:09.442874    2578 generic.go:356] "Generic (PLEG): container finished" podID="285dc3dd-5faf-480b-9629-a6f7dc0c4b24" containerID="5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb" exitCode=0
Sep 09 12:43:09.443089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:09.442924    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"
Sep 09 12:43:09.443089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:09.442968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4" event={"ID":"285dc3dd-5faf-480b-9629-a6f7dc0c4b24","Type":"ContainerDied","Data":"5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb"}
Sep 09 12:43:09.443089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:09.443010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4" event={"ID":"285dc3dd-5faf-480b-9629-a6f7dc0c4b24","Type":"ContainerDied","Data":"4d6e536fa01e36ff347c3abaf61842314e867999d514a736fa45f53dd1ff9b2e"}
Sep 09 12:43:09.443089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:09.443027    2578 scope.go:117] "RemoveContainer" containerID="5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb"
Sep 09 12:43:09.452663 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:09.452646    2578 scope.go:117] "RemoveContainer" containerID="5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb"
Sep 09 12:43:09.452950 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:43:09.452933    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb\": container with ID starting with 5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb not found: ID does not exist" containerID="5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb"
Sep 09 12:43:09.453000 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:09.452960    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb"} err="failed to get container status \"5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb\": rpc error: code = NotFound desc = could not find container \"5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb\": container with ID starting with 5142275b181ea40a7421c2be5c474a54c7782f0b6378f7f9c60b6592d62bbcdb not found: ID does not exist"
Sep 09 12:43:09.466708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:09.466670    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"]
Sep 09 12:43:09.470826 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:09.470799    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4"]
Sep 09 12:43:10.186861 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:10.186824    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285dc3dd-5faf-480b-9629-a6f7dc0c4b24" path="/var/lib/kubelet/pods/285dc3dd-5faf-480b-9629-a6f7dc0c4b24/volumes"
Sep 09 12:43:13.646169 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:13.646131    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fe6606/auth-test-model-fe6606-kserve-7d9f8c78cf-xhrd4" podUID="285dc3dd-5faf-480b-9629-a6f7dc0c4b24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 12:43:15.752891 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:15.752852    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:15.753396 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:15.752913    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:15.765395 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:15.765365    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:16.490541 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:16.490514    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:43:34.196524 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.196442    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"]
Sep 09 12:43:34.197106 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.197086    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285dc3dd-5faf-480b-9629-a6f7dc0c4b24" containerName="main"
Sep 09 12:43:34.197183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.197110    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="285dc3dd-5faf-480b-9629-a6f7dc0c4b24" containerName="main"
Sep 09 12:43:34.197280 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.197267    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="285dc3dd-5faf-480b-9629-a6f7dc0c4b24" containerName="main"
Sep 09 12:43:34.202393 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.202370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.204345 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.204324    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ab8d88-serving-cert\""
Sep 09 12:43:34.213366 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.212780    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"]
Sep 09 12:43:34.357541 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.357503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-metrics-certs\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.357541 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.357540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-maas-api-tls\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.357743 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.357562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20a9482-46ad-49a2-864c-ab4fbc548d32-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.357743 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.357589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxcp9\" (UniqueName: \"kubernetes.io/projected/f20a9482-46ad-49a2-864c-ab4fbc548d32-kube-api-access-vxcp9\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.458846 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.458756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-metrics-certs\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.458846 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.458799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-maas-api-tls\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.458846 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.458820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20a9482-46ad-49a2-864c-ab4fbc548d32-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.459136 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.458859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxcp9\" (UniqueName: \"kubernetes.io/projected/f20a9482-46ad-49a2-864c-ab4fbc548d32-kube-api-access-vxcp9\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.459136 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:43:34.458947    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-serving-cert: secret "maas-api-e2e-api-a-ab8d88-serving-cert" not found
Sep 09 12:43:34.459136 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:43:34.459029    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-maas-api-tls podName:f20a9482-46ad-49a2-864c-ab4fbc548d32 nodeName:}" failed. No retries permitted until 2026-09-09 12:43:34.959011431 +0000 UTC m=+1435.302609357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-maas-api-tls") pod "maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99" (UID: "f20a9482-46ad-49a2-864c-ab4fbc548d32") : secret "maas-api-e2e-api-a-ab8d88-serving-cert" not found
Sep 09 12:43:34.459581 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.459553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20a9482-46ad-49a2-864c-ab4fbc548d32-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.461406 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.461385    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-metrics-certs\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.468952 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.468924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxcp9\" (UniqueName: \"kubernetes.io/projected/f20a9482-46ad-49a2-864c-ab4fbc548d32-kube-api-access-vxcp9\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.898577 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.898496    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"]
Sep 09 12:43:34.902269 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.902249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:43:34.904386 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.904360    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ab8d88-dockercfg-7sj5w\""
Sep 09 12:43:34.904487 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.904472    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ab8d88\""
Sep 09 12:43:34.913653 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.913629    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"]
Sep 09 12:43:34.964393 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.964364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-maas-api-tls\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:34.966871 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:34.966843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-maas-api-tls\") pod \"maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:35.065386 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.065343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d4aab6-23d0-49f3-9298-730cafe22208-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p\" (UID: \"52d4aab6-23d0-49f3-9298-730cafe22208\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:43:35.065562 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.065404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t84rq\" (UniqueName: \"kubernetes.io/projected/52d4aab6-23d0-49f3-9298-730cafe22208-kube-api-access-t84rq\") pod \"payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p\" (UID: \"52d4aab6-23d0-49f3-9298-730cafe22208\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:43:35.117402 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.117361    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:35.167108 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.167044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d4aab6-23d0-49f3-9298-730cafe22208-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p\" (UID: \"52d4aab6-23d0-49f3-9298-730cafe22208\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:43:35.167284 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.167132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t84rq\" (UniqueName: \"kubernetes.io/projected/52d4aab6-23d0-49f3-9298-730cafe22208-kube-api-access-t84rq\") pod \"payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p\" (UID: \"52d4aab6-23d0-49f3-9298-730cafe22208\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:43:35.168386 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.168358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d4aab6-23d0-49f3-9298-730cafe22208-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p\" (UID: \"52d4aab6-23d0-49f3-9298-730cafe22208\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:43:35.175352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.175328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t84rq\" (UniqueName: \"kubernetes.io/projected/52d4aab6-23d0-49f3-9298-730cafe22208-kube-api-access-t84rq\") pod \"payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p\" (UID: \"52d4aab6-23d0-49f3-9298-730cafe22208\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:43:35.213803 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.213775    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:43:35.394387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.394362    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"]
Sep 09 12:43:35.396028 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:43:35.396002    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20a9482_46ad_49a2_864c_ab4fbc548d32.slice/crio-a2c476fadd08d2816f040cc4721dd3d8f1f23776e4a6e00844ef5aced9584e61 WatchSource:0}: Error finding container a2c476fadd08d2816f040cc4721dd3d8f1f23776e4a6e00844ef5aced9584e61: Status 404 returned error can't find the container with id a2c476fadd08d2816f040cc4721dd3d8f1f23776e4a6e00844ef5aced9584e61
Sep 09 12:43:35.397380 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.397360    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:43:35.487151 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.487125    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"]
Sep 09 12:43:35.489075 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:43:35.489048    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d4aab6_23d0_49f3_9298_730cafe22208.slice/crio-5982b53896154abcc8156575d172462cfb1374a7ccddcfef96e4dc6f744caf28 WatchSource:0}: Error finding container 5982b53896154abcc8156575d172462cfb1374a7ccddcfef96e4dc6f744caf28: Status 404 returned error can't find the container with id 5982b53896154abcc8156575d172462cfb1374a7ccddcfef96e4dc6f744caf28
Sep 09 12:43:35.581914 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.581889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99" event={"ID":"f20a9482-46ad-49a2-864c-ab4fbc548d32","Type":"ContainerStarted","Data":"a2c476fadd08d2816f040cc4721dd3d8f1f23776e4a6e00844ef5aced9584e61"}
Sep 09 12:43:35.583674 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.583646    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p" event={"ID":"52d4aab6-23d0-49f3-9298-730cafe22208","Type":"ContainerStarted","Data":"088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e"}
Sep 09 12:43:35.583792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.583700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p" event={"ID":"52d4aab6-23d0-49f3-9298-730cafe22208","Type":"ContainerStarted","Data":"5982b53896154abcc8156575d172462cfb1374a7ccddcfef96e4dc6f744caf28"}
Sep 09 12:43:35.583792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.583782    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:43:35.601021 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:35.600979    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p" podStartSLOduration=1.600964967 podStartE2EDuration="1.600964967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:43:35.599109578 +0000 UTC m=+1435.942707530" watchObservedRunningTime="2026-09-09 12:43:35.600964967 +0000 UTC m=+1435.944562907"
Sep 09 12:43:36.590419 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:36.590375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99" event={"ID":"f20a9482-46ad-49a2-864c-ab4fbc548d32","Type":"ContainerStarted","Data":"008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a"}
Sep 09 12:43:36.590836 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:36.590535    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:36.610746 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:36.610689    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99" podStartSLOduration=2.234278816 podStartE2EDuration="2.610673871s" totalImagesPullingTime="376.395055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:43:36.607988334 +0000 UTC m=+1436.951586295" watchObservedRunningTime="2026-09-09 12:43:36.610673871 +0000 UTC m=+1436.954271818"
Sep 09 12:43:42.600580 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:42.600552    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:43:43.232150 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:43.232113    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:43:46.592856 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:43:46.592820    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:44:03.419474 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.419444    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"]
Sep 09 12:44:03.424495 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.424467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:44:03.426857 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.426831    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3f4a7d\""
Sep 09 12:44:03.427004 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.426832    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3f4a7d-dockercfg-gbb9l\""
Sep 09 12:44:03.436132 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.436111    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"]
Sep 09 12:44:03.556864 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.556829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b78815-223c-48af-94af-2d5a776065b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb\" (UID: \"37b78815-223c-48af-94af-2d5a776065b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:44:03.557041 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.556889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nlj\" (UniqueName: \"kubernetes.io/projected/37b78815-223c-48af-94af-2d5a776065b5-kube-api-access-w9nlj\") pod \"payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb\" (UID: \"37b78815-223c-48af-94af-2d5a776065b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:44:03.658197 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.658163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nlj\" (UniqueName: \"kubernetes.io/projected/37b78815-223c-48af-94af-2d5a776065b5-kube-api-access-w9nlj\") pod \"payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb\" (UID: \"37b78815-223c-48af-94af-2d5a776065b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:44:03.658408 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.658328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b78815-223c-48af-94af-2d5a776065b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb\" (UID: \"37b78815-223c-48af-94af-2d5a776065b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:44:03.659020 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.658999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b78815-223c-48af-94af-2d5a776065b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb\" (UID: \"37b78815-223c-48af-94af-2d5a776065b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:44:03.666913 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.666883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nlj\" (UniqueName: \"kubernetes.io/projected/37b78815-223c-48af-94af-2d5a776065b5-kube-api-access-w9nlj\") pod \"payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb\" (UID: \"37b78815-223c-48af-94af-2d5a776065b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:44:03.735684 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:03.735601    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:44:04.017841 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:04.017815    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"]
Sep 09 12:44:04.019501 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:44:04.019473    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b78815_223c_48af_94af_2d5a776065b5.slice/crio-9bb0d5cb920f51e3bf1f0cc0994dac2586d1bebf5e0d81ef4974d606fb1b7752 WatchSource:0}: Error finding container 9bb0d5cb920f51e3bf1f0cc0994dac2586d1bebf5e0d81ef4974d606fb1b7752: Status 404 returned error can't find the container with id 9bb0d5cb920f51e3bf1f0cc0994dac2586d1bebf5e0d81ef4974d606fb1b7752
Sep 09 12:44:04.743564 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:04.743526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb" event={"ID":"37b78815-223c-48af-94af-2d5a776065b5","Type":"ContainerStarted","Data":"e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d"}
Sep 09 12:44:04.743990 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:04.743625    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:44:04.743990 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:04.743643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb" event={"ID":"37b78815-223c-48af-94af-2d5a776065b5","Type":"ContainerStarted","Data":"9bb0d5cb920f51e3bf1f0cc0994dac2586d1bebf5e0d81ef4974d606fb1b7752"}
Sep 09 12:44:04.763721 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:04.763671    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb" podStartSLOduration=1.763655957 podStartE2EDuration="1.763655957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:44:04.761239518 +0000 UTC m=+1465.104837473" watchObservedRunningTime="2026-09-09 12:44:04.763655957 +0000 UTC m=+1465.107253912"
Sep 09 12:44:11.434323 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:11.434281    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:44:15.750602 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:15.750573    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:44:27.052732 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:27.052697    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:44:29.879403 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:29.879372    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7675-269aa5"]
Sep 09 12:44:29.883083 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:29.883067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-269aa5"
Sep 09 12:44:29.885092 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:29.885061    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4dspl\""
Sep 09 12:44:29.890909 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:29.890806    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7675-269aa5"]
Sep 09 12:44:29.928015 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:29.927982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7q4j\" (UniqueName: \"kubernetes.io/projected/39549b01-8248-4616-957e-0c8c38dd0a1f-kube-api-access-t7q4j\") pod \"test-curl-7675-269aa5\" (UID: \"39549b01-8248-4616-957e-0c8c38dd0a1f\") " pod="opendatahub/test-curl-7675-269aa5"
Sep 09 12:44:30.029474 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:30.029434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7q4j\" (UniqueName: \"kubernetes.io/projected/39549b01-8248-4616-957e-0c8c38dd0a1f-kube-api-access-t7q4j\") pod \"test-curl-7675-269aa5\" (UID: \"39549b01-8248-4616-957e-0c8c38dd0a1f\") " pod="opendatahub/test-curl-7675-269aa5"
Sep 09 12:44:30.037871 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:30.037833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7q4j\" (UniqueName: \"kubernetes.io/projected/39549b01-8248-4616-957e-0c8c38dd0a1f-kube-api-access-t7q4j\") pod \"test-curl-7675-269aa5\" (UID: \"39549b01-8248-4616-957e-0c8c38dd0a1f\") " pod="opendatahub/test-curl-7675-269aa5"
Sep 09 12:44:30.193929 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:30.193899    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-269aa5"
Sep 09 12:44:30.469633 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:30.469611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7675-269aa5"]
Sep 09 12:44:30.472047 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:44:30.472020    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39549b01_8248_4616_957e_0c8c38dd0a1f.slice/crio-da68020477cceb0070a716764d068bc3b6abff3e267b69292aab17b9a49ddb36 WatchSource:0}: Error finding container da68020477cceb0070a716764d068bc3b6abff3e267b69292aab17b9a49ddb36: Status 404 returned error can't find the container with id da68020477cceb0070a716764d068bc3b6abff3e267b69292aab17b9a49ddb36
Sep 09 12:44:30.883354 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:30.883268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7675-269aa5" event={"ID":"39549b01-8248-4616-957e-0c8c38dd0a1f","Type":"ContainerStarted","Data":"619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df"}
Sep 09 12:44:30.883354 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:30.883312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7675-269aa5" event={"ID":"39549b01-8248-4616-957e-0c8c38dd0a1f","Type":"ContainerStarted","Data":"da68020477cceb0070a716764d068bc3b6abff3e267b69292aab17b9a49ddb36"}
Sep 09 12:44:30.900126 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:30.900077    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7675-269aa5" podStartSLOduration=1.900061469 podStartE2EDuration="1.900061469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:44:30.897853972 +0000 UTC m=+1491.241451920" watchObservedRunningTime="2026-09-09 12:44:30.900061469 +0000 UTC m=+1491.243659463"
Sep 09 12:44:31.543254 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.543205    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7675-269aa5"]
Sep 09 12:44:31.547263 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.547204    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7675-269aa5"]
Sep 09 12:44:31.660705 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.660676    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7675-1cf214"]
Sep 09 12:44:31.661135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.661122    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39549b01-8248-4616-957e-0c8c38dd0a1f" containerName="test-curl-7675-269aa5"
Sep 09 12:44:31.661189 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.661136    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="39549b01-8248-4616-957e-0c8c38dd0a1f" containerName="test-curl-7675-269aa5"
Sep 09 12:44:31.661255 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.661201    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="39549b01-8248-4616-957e-0c8c38dd0a1f" containerName="test-curl-7675-269aa5"
Sep 09 12:44:31.664423 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.664409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-1cf214"
Sep 09 12:44:31.671536 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.671449    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7675-1cf214"]
Sep 09 12:44:31.748104 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.748066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8z7\" (UniqueName: \"kubernetes.io/projected/92c2c75f-9596-439d-94dd-07b5d9df1664-kube-api-access-4f8z7\") pod \"test-curl-7675-1cf214\" (UID: \"92c2c75f-9596-439d-94dd-07b5d9df1664\") " pod="opendatahub/test-curl-7675-1cf214"
Sep 09 12:44:31.849329 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.849296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8z7\" (UniqueName: \"kubernetes.io/projected/92c2c75f-9596-439d-94dd-07b5d9df1664-kube-api-access-4f8z7\") pod \"test-curl-7675-1cf214\" (UID: \"92c2c75f-9596-439d-94dd-07b5d9df1664\") " pod="opendatahub/test-curl-7675-1cf214"
Sep 09 12:44:31.857277 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.857249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8z7\" (UniqueName: \"kubernetes.io/projected/92c2c75f-9596-439d-94dd-07b5d9df1664-kube-api-access-4f8z7\") pod \"test-curl-7675-1cf214\" (UID: \"92c2c75f-9596-439d-94dd-07b5d9df1664\") " pod="opendatahub/test-curl-7675-1cf214"
Sep 09 12:44:31.888809 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:44:31.888785    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7675-269aa5" container="test-curl-7675-269aa5"
Sep 09 12:44:31.974805 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:31.974726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-1cf214"
Sep 09 12:44:32.248356 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:32.248329    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7675-1cf214"]
Sep 09 12:44:32.250358 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:44:32.250336    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92c2c75f_9596_439d_94dd_07b5d9df1664.slice/crio-482395d524d1be00323859b32f62f332494a5a2ef46e3fa80935555d636da0f2 WatchSource:0}: Error finding container 482395d524d1be00323859b32f62f332494a5a2ef46e3fa80935555d636da0f2: Status 404 returned error can't find the container with id 482395d524d1be00323859b32f62f332494a5a2ef46e3fa80935555d636da0f2
Sep 09 12:44:32.895892 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:32.895853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7675-1cf214" event={"ID":"92c2c75f-9596-439d-94dd-07b5d9df1664","Type":"ContainerStarted","Data":"8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91"}
Sep 09 12:44:32.896371 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:32.895901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7675-1cf214" event={"ID":"92c2c75f-9596-439d-94dd-07b5d9df1664","Type":"ContainerStarted","Data":"482395d524d1be00323859b32f62f332494a5a2ef46e3fa80935555d636da0f2"}
Sep 09 12:44:32.896371 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:32.895959    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7675-269aa5" podUID="39549b01-8248-4616-957e-0c8c38dd0a1f" containerName="test-curl-7675-269aa5" containerID="cri-o://619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df" gracePeriod=2
Sep 09 12:44:32.912983 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:32.912935    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7675-1cf214" podStartSLOduration=1.912919446 podStartE2EDuration="1.912919446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:44:32.9107367 +0000 UTC m=+1493.254334648" watchObservedRunningTime="2026-09-09 12:44:32.912919446 +0000 UTC m=+1493.256517414"
Sep 09 12:44:33.495448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:33.495407    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7675-1cf214"]
Sep 09 12:44:33.499670 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:33.499644    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7675-1cf214"]
Sep 09 12:44:33.901823 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:44:33.901791    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7675-1cf214" container="test-curl-7675-1cf214"
Sep 09 12:44:34.907665 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:34.907631    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7675-1cf214" podUID="92c2c75f-9596-439d-94dd-07b5d9df1664" containerName="test-curl-7675-1cf214" containerID="cri-o://8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91" gracePeriod=2
Sep 09 12:44:35.084130 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:35.084103    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-269aa5"
Sep 09 12:44:35.182442 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:35.182410    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7q4j\" (UniqueName: \"kubernetes.io/projected/39549b01-8248-4616-957e-0c8c38dd0a1f-kube-api-access-t7q4j\") pod \"39549b01-8248-4616-957e-0c8c38dd0a1f\" (UID: \"39549b01-8248-4616-957e-0c8c38dd0a1f\") "
Sep 09 12:44:35.184615 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:35.184585    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39549b01-8248-4616-957e-0c8c38dd0a1f-kube-api-access-t7q4j" (OuterVolumeSpecName: "kube-api-access-t7q4j") pod "39549b01-8248-4616-957e-0c8c38dd0a1f" (UID: "39549b01-8248-4616-957e-0c8c38dd0a1f"). InnerVolumeSpecName "kube-api-access-t7q4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:44:35.283697 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:35.283658    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7q4j\" (UniqueName: \"kubernetes.io/projected/39549b01-8248-4616-957e-0c8c38dd0a1f-kube-api-access-t7q4j\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:35.552964 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:35.552882    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"]
Sep 09 12:44:35.913488 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:35.913452    2578 generic.go:356] "Generic (PLEG): container finished" podID="39549b01-8248-4616-957e-0c8c38dd0a1f" containerID="619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df" exitCode=137
Sep 09 12:44:35.913915 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:35.913505    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-269aa5"
Sep 09 12:44:35.913915 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:35.913499    2578 scope.go:117] "RemoveContainer" containerID="619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df"
Sep 09 12:44:35.922948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:35.922927    2578 scope.go:117] "RemoveContainer" containerID="619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df"
Sep 09 12:44:35.923175 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:44:35.923147    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7675-269aa5\" is forbidden: User \"system:node:ip-10-0-133-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-152.ec2.internal' and this object" podUID="39549b01-8248-4616-957e-0c8c38dd0a1f" pod="opendatahub/test-curl-7675-269aa5"
Sep 09 12:44:35.923280 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:44:35.923241    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df\": container with ID starting with 619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df not found: ID does not exist" containerID="619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df"
Sep 09 12:44:35.923280 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:35.923267    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df"} err="failed to get container status \"619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df\": rpc error: code = NotFound desc = could not find container \"619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df\": container with ID starting with 619b21f09f3907bed01ad35ddbb1bb1d3ca05a4da90f55e114290e740ca348df not found: ID does not exist"
Sep 09 12:44:36.186543 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:36.186462    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39549b01-8248-4616-957e-0c8c38dd0a1f" path="/var/lib/kubelet/pods/39549b01-8248-4616-957e-0c8c38dd0a1f/volumes"
Sep 09 12:44:37.087777 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:37.087749    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-1cf214"
Sep 09 12:44:37.204458 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:37.204365    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f8z7\" (UniqueName: \"kubernetes.io/projected/92c2c75f-9596-439d-94dd-07b5d9df1664-kube-api-access-4f8z7\") pod \"92c2c75f-9596-439d-94dd-07b5d9df1664\" (UID: \"92c2c75f-9596-439d-94dd-07b5d9df1664\") "
Sep 09 12:44:37.206616 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:37.206583    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c2c75f-9596-439d-94dd-07b5d9df1664-kube-api-access-4f8z7" (OuterVolumeSpecName: "kube-api-access-4f8z7") pod "92c2c75f-9596-439d-94dd-07b5d9df1664" (UID: "92c2c75f-9596-439d-94dd-07b5d9df1664"). InnerVolumeSpecName "kube-api-access-4f8z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:44:37.305666 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:37.305613    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f8z7\" (UniqueName: \"kubernetes.io/projected/92c2c75f-9596-439d-94dd-07b5d9df1664-kube-api-access-4f8z7\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:37.925819 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:37.925788    2578 generic.go:356] "Generic (PLEG): container finished" podID="92c2c75f-9596-439d-94dd-07b5d9df1664" containerID="8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91" exitCode=137
Sep 09 12:44:37.925991 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:37.925836    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7675-1cf214"
Sep 09 12:44:37.925991 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:37.925879    2578 scope.go:117] "RemoveContainer" containerID="8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91"
Sep 09 12:44:37.935080 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:37.935061    2578 scope.go:117] "RemoveContainer" containerID="8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91"
Sep 09 12:44:37.935153 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:44:37.935057    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7675-1cf214\" is forbidden: User \"system:node:ip-10-0-133-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-152.ec2.internal' and this object" podUID="92c2c75f-9596-439d-94dd-07b5d9df1664" pod="opendatahub/test-curl-7675-1cf214"
Sep 09 12:44:37.935424 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:44:37.935403    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91\": container with ID starting with 8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91 not found: ID does not exist" containerID="8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91"
Sep 09 12:44:37.935501 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:37.935435    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91"} err="failed to get container status \"8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91\": rpc error: code = NotFound desc = could not find container \"8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91\": container with ID starting with 8bc7f65d8fee06abbe6c1834834a271776fdbee821d9fd85ce9a122abebbfb91 not found: ID does not exist"
Sep 09 12:44:38.186408 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:38.186331    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92c2c75f-9596-439d-94dd-07b5d9df1664" path="/var/lib/kubelet/pods/92c2c75f-9596-439d-94dd-07b5d9df1664/volumes"
Sep 09 12:44:40.254680 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:40.254648    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:44:40.262446 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:40.262420    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:44:45.560740 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.560701    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"]
Sep 09 12:44:45.561363 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.560969    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99" podUID="f20a9482-46ad-49a2-864c-ab4fbc548d32" containerName="maas-api" containerID="cri-o://008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a" gracePeriod=30
Sep 09 12:44:45.620571 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.620532    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"]
Sep 09 12:44:45.620793 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.620769    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p" podUID="52d4aab6-23d0-49f3-9298-730cafe22208" containerName="payload-processing" containerID="cri-o://088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e" gracePeriod=30
Sep 09 12:44:45.871242 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.871190    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:44:45.899327 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.899287    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxcp9\" (UniqueName: \"kubernetes.io/projected/f20a9482-46ad-49a2-864c-ab4fbc548d32-kube-api-access-vxcp9\") pod \"f20a9482-46ad-49a2-864c-ab4fbc548d32\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") "
Sep 09 12:44:45.899479 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.899347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20a9482-46ad-49a2-864c-ab4fbc548d32-openshift-service-ca\") pod \"f20a9482-46ad-49a2-864c-ab4fbc548d32\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") "
Sep 09 12:44:45.899479 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.899393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-maas-api-tls\") pod \"f20a9482-46ad-49a2-864c-ab4fbc548d32\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") "
Sep 09 12:44:45.899479 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.899446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-metrics-certs\") pod \"f20a9482-46ad-49a2-864c-ab4fbc548d32\" (UID: \"f20a9482-46ad-49a2-864c-ab4fbc548d32\") "
Sep 09 12:44:45.899976 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.899950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20a9482-46ad-49a2-864c-ab4fbc548d32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f20a9482-46ad-49a2-864c-ab4fbc548d32" (UID: "f20a9482-46ad-49a2-864c-ab4fbc548d32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:44:45.902017 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.901993    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20a9482-46ad-49a2-864c-ab4fbc548d32-kube-api-access-vxcp9" (OuterVolumeSpecName: "kube-api-access-vxcp9") pod "f20a9482-46ad-49a2-864c-ab4fbc548d32" (UID: "f20a9482-46ad-49a2-864c-ab4fbc548d32"). InnerVolumeSpecName "kube-api-access-vxcp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:44:45.902175 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.902147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f20a9482-46ad-49a2-864c-ab4fbc548d32" (UID: "f20a9482-46ad-49a2-864c-ab4fbc548d32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:44:45.902435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.902413    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f20a9482-46ad-49a2-864c-ab4fbc548d32" (UID: "f20a9482-46ad-49a2-864c-ab4fbc548d32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:44:45.905424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.905407    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:44:45.969422 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.969387    2578 generic.go:356] "Generic (PLEG): container finished" podID="f20a9482-46ad-49a2-864c-ab4fbc548d32" containerID="008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a" exitCode=0
Sep 09 12:44:45.969596 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.969435    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"
Sep 09 12:44:45.969596 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.969475    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99" event={"ID":"f20a9482-46ad-49a2-864c-ab4fbc548d32","Type":"ContainerDied","Data":"008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a"}
Sep 09 12:44:45.969596 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.969537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99" event={"ID":"f20a9482-46ad-49a2-864c-ab4fbc548d32","Type":"ContainerDied","Data":"a2c476fadd08d2816f040cc4721dd3d8f1f23776e4a6e00844ef5aced9584e61"}
Sep 09 12:44:45.969596 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.969568    2578 scope.go:117] "RemoveContainer" containerID="008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a"
Sep 09 12:44:45.971291 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.971249    2578 generic.go:356] "Generic (PLEG): container finished" podID="52d4aab6-23d0-49f3-9298-730cafe22208" containerID="088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e" exitCode=0
Sep 09 12:44:45.971408 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.971295    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"
Sep 09 12:44:45.971408 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.971283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p" event={"ID":"52d4aab6-23d0-49f3-9298-730cafe22208","Type":"ContainerDied","Data":"088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e"}
Sep 09 12:44:45.971408 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.971349    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p" event={"ID":"52d4aab6-23d0-49f3-9298-730cafe22208","Type":"ContainerDied","Data":"5982b53896154abcc8156575d172462cfb1374a7ccddcfef96e4dc6f744caf28"}
Sep 09 12:44:45.979626 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.979608    2578 scope.go:117] "RemoveContainer" containerID="008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a"
Sep 09 12:44:45.979950 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:44:45.979930    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a\": container with ID starting with 008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a not found: ID does not exist" containerID="008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a"
Sep 09 12:44:45.980010 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.979959    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a"} err="failed to get container status \"008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a\": rpc error: code = NotFound desc = could not find container \"008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a\": container with ID starting with 008f22c5d539a6bbf31642029db89bb99c8b04f77f167926acf0f481f40ca26a not found: ID does not exist"
Sep 09 12:44:45.980010 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.979975    2578 scope.go:117] "RemoveContainer" containerID="088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e"
Sep 09 12:44:45.988729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.988712    2578 scope.go:117] "RemoveContainer" containerID="088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e"
Sep 09 12:44:45.989097 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:44:45.989069    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e\": container with ID starting with 088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e not found: ID does not exist" containerID="088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e"
Sep 09 12:44:45.989230 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.989110    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e"} err="failed to get container status \"088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e\": rpc error: code = NotFound desc = could not find container \"088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e\": container with ID starting with 088d6bdd32637bd149843d413d145d2a8497888cfe62e59ca8d9b8d683735c3e not found: ID does not exist"
Sep 09 12:44:45.991478 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.991457    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"]
Sep 09 12:44:45.995146 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:45.995121    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab8d88-76c549fdd5-lrb99"]
Sep 09 12:44:46.000156 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.000138    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t84rq\" (UniqueName: \"kubernetes.io/projected/52d4aab6-23d0-49f3-9298-730cafe22208-kube-api-access-t84rq\") pod \"52d4aab6-23d0-49f3-9298-730cafe22208\" (UID: \"52d4aab6-23d0-49f3-9298-730cafe22208\") "
Sep 09 12:44:46.000330 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.000310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d4aab6-23d0-49f3-9298-730cafe22208-plugins-config-volume\") pod \"52d4aab6-23d0-49f3-9298-730cafe22208\" (UID: \"52d4aab6-23d0-49f3-9298-730cafe22208\") "
Sep 09 12:44:46.000550 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.000536    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxcp9\" (UniqueName: \"kubernetes.io/projected/f20a9482-46ad-49a2-864c-ab4fbc548d32-kube-api-access-vxcp9\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:46.000613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.000552    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20a9482-46ad-49a2-864c-ab4fbc548d32-openshift-service-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:46.000613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.000562    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-maas-api-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:46.000613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.000570    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f20a9482-46ad-49a2-864c-ab4fbc548d32-metrics-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:46.000718 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.000629    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d4aab6-23d0-49f3-9298-730cafe22208-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52d4aab6-23d0-49f3-9298-730cafe22208" (UID: "52d4aab6-23d0-49f3-9298-730cafe22208"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:44:46.002292 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.002276    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d4aab6-23d0-49f3-9298-730cafe22208-kube-api-access-t84rq" (OuterVolumeSpecName: "kube-api-access-t84rq") pod "52d4aab6-23d0-49f3-9298-730cafe22208" (UID: "52d4aab6-23d0-49f3-9298-730cafe22208"). InnerVolumeSpecName "kube-api-access-t84rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:44:46.101453 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.101359    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d4aab6-23d0-49f3-9298-730cafe22208-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:46.101453 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.101395    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t84rq\" (UniqueName: \"kubernetes.io/projected/52d4aab6-23d0-49f3-9298-730cafe22208-kube-api-access-t84rq\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:46.186903 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.186873    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20a9482-46ad-49a2-864c-ab4fbc548d32" path="/var/lib/kubelet/pods/f20a9482-46ad-49a2-864c-ab4fbc548d32/volumes"
Sep 09 12:44:46.286741 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.286709    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"]
Sep 09 12:44:46.290343 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:46.290318    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab8d88-7997c466c7-tt72p"]
Sep 09 12:44:48.186517 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:48.186484    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52d4aab6-23d0-49f3-9298-730cafe22208" path="/var/lib/kubelet/pods/52d4aab6-23d0-49f3-9298-730cafe22208/volumes"
Sep 09 12:44:49.834003 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:49.833961    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:44:50.572878 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.572845    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9" podUID="ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" containerName="main" containerID="cri-o://e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d" gracePeriod=45
Sep 09 12:44:50.827047 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.826985    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:44:50.842175 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842152    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-dshm\") pod \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") "
Sep 09 12:44:50.842590 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842194    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-model-cache\") pod \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") "
Sep 09 12:44:50.842590 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842277    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tls-certs\") pod \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") "
Sep 09 12:44:50.842590 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842335    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmtnb\" (UniqueName: \"kubernetes.io/projected/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-kube-api-access-mmtnb\") pod \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") "
Sep 09 12:44:50.842590 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842383    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-home\") pod \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") "
Sep 09 12:44:50.842590 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842422    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tmp-dir\") pod \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\" (UID: \"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c\") "
Sep 09 12:44:50.842590 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" (UID: "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:44:50.842927 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842631    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-home" (OuterVolumeSpecName: "home") pod "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" (UID: "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:44:50.842927 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" (UID: "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:44:50.842927 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842785    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-home\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:50.842927 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842866    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tmp-dir\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:50.843079 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.842983    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-model-cache\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:50.845143 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.845115    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" (UID: "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:44:50.845304 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.845251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-dshm" (OuterVolumeSpecName: "dshm") pod "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" (UID: "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:44:50.845380 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.845334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-kube-api-access-mmtnb" (OuterVolumeSpecName: "kube-api-access-mmtnb") pod "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" (UID: "ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c"). InnerVolumeSpecName "kube-api-access-mmtnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:44:50.943993 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.943950    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-dshm\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:50.943993 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.943979    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-tls-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:50.943993 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:50.943995    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmtnb\" (UniqueName: \"kubernetes.io/projected/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c-kube-api-access-mmtnb\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:44:51.001552 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:51.001519    2578 generic.go:356] "Generic (PLEG): container finished" podID="ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" containerID="e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d" exitCode=0
Sep 09 12:44:51.001714 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:51.001581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9" event={"ID":"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c","Type":"ContainerDied","Data":"e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d"}
Sep 09 12:44:51.001714 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:51.001609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9" event={"ID":"ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c","Type":"ContainerDied","Data":"3fcbd73edd2915e6b71b159ea2a69e3a3fecf0e66c82b35520c04a01d12145f1"}
Sep 09 12:44:51.001714 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:51.001625    2578 scope.go:117] "RemoveContainer" containerID="e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d"
Sep 09 12:44:51.001825 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:51.001727    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"
Sep 09 12:44:51.017954 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:51.017931    2578 scope.go:117] "RemoveContainer" containerID="e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d"
Sep 09 12:44:51.018296 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:44:51.018265    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d\": container with ID starting with e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d not found: ID does not exist" containerID="e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d"
Sep 09 12:44:51.018391 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:51.018302    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d"} err="failed to get container status \"e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d\": rpc error: code = NotFound desc = could not find container \"e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d\": container with ID starting with e48f390d53d06f5f45ecb5859da758c6d107beb779acd91fb12815f2c4de782d not found: ID does not exist"
Sep 09 12:44:51.032678 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:51.032654    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"]
Sep 09 12:44:51.037403 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:51.037383    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-283adb/sub-test-model-283adb-kserve-67477d6c87-qxsq9"]
Sep 09 12:44:52.186409 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:44:52.186381    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" path="/var/lib/kubelet/pods/ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c/volumes"
Sep 09 12:45:00.154621 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.154587    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb"]
Sep 09 12:45:00.163664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.163622    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d4aab6-23d0-49f3-9298-730cafe22208" containerName="payload-processing"
Sep 09 12:45:00.163664 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.163656    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d4aab6-23d0-49f3-9298-730cafe22208" containerName="payload-processing"
Sep 09 12:45:00.163853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.163678    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20a9482-46ad-49a2-864c-ab4fbc548d32" containerName="maas-api"
Sep 09 12:45:00.163853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.163686    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20a9482-46ad-49a2-864c-ab4fbc548d32" containerName="maas-api"
Sep 09 12:45:00.163853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.163699    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" containerName="main"
Sep 09 12:45:00.163853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.163706    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" containerName="main"
Sep 09 12:45:00.163853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.163742    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c2c75f-9596-439d-94dd-07b5d9df1664" containerName="test-curl-7675-1cf214"
Sep 09 12:45:00.163853 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.163748    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c2c75f-9596-439d-94dd-07b5d9df1664" containerName="test-curl-7675-1cf214"
Sep 09 12:45:00.164039 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.163993    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3c2c1a-bce3-4e30-9b71-7ad5f8b2303c" containerName="main"
Sep 09 12:45:00.164039 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.164018    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d4aab6-23d0-49f3-9298-730cafe22208" containerName="payload-processing"
Sep 09 12:45:00.164039 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.164032    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c2c75f-9596-439d-94dd-07b5d9df1664" containerName="test-curl-7675-1cf214"
Sep 09 12:45:00.164132 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.164044    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20a9482-46ad-49a2-864c-ab4fbc548d32" containerName="maas-api"
Sep 09 12:45:00.169303 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.169276    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj"]
Sep 09 12:45:00.169464 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.169438    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb"
Sep 09 12:45:00.172075 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.172053    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wvr9\""
Sep 09 12:45:00.175294 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.175273    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj"]
Sep 09 12:45:00.175468 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.175454    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb"]
Sep 09 12:45:00.175652 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.175638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj"
Sep 09 12:45:00.184481 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.184463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb"
Sep 09 12:45:00.189060 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.189010    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg"]
Sep 09 12:45:00.193971 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.193932    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg"
Sep 09 12:45:00.194854 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.194836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj"
Sep 09 12:45:00.197146 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.197117    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg"]
Sep 09 12:45:00.240833 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.240805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg"
Sep 09 12:45:00.481337 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.481312    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb"]
Sep 09 12:45:00.483988 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:45:00.483958    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea9daa7_5a52_4dda_888f_d7256983c5cb.slice/crio-0b34aee17d290b892e0fdcc37ea29dbcb7604be2c9d71fbce0cd1a9faaecc30f WatchSource:0}: Error finding container 0b34aee17d290b892e0fdcc37ea29dbcb7604be2c9d71fbce0cd1a9faaecc30f: Status 404 returned error can't find the container with id 0b34aee17d290b892e0fdcc37ea29dbcb7604be2c9d71fbce0cd1a9faaecc30f
Sep 09 12:45:00.484669 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.484647    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj"]
Sep 09 12:45:00.486597 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:45:00.486580    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cef956_7266_4c66_a0ef_28366e6670ec.slice/crio-655feea374e58bf8ea113f16cf3cd1601f9a2dd7a01a8775db5056f669fde7a7 WatchSource:0}: Error finding container 655feea374e58bf8ea113f16cf3cd1601f9a2dd7a01a8775db5056f669fde7a7: Status 404 returned error can't find the container with id 655feea374e58bf8ea113f16cf3cd1601f9a2dd7a01a8775db5056f669fde7a7
Sep 09 12:45:00.544715 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:00.544687    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg"]
Sep 09 12:45:00.546842 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:45:00.546819    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef22df4_2239_4a4d_8adf_da77f3ae503f.slice/crio-5a220b41406baf09ce33339e3c3f76e02097225a7240d6db7d1a247c194138d7 WatchSource:0}: Error finding container 5a220b41406baf09ce33339e3c3f76e02097225a7240d6db7d1a247c194138d7: Status 404 returned error can't find the container with id 5a220b41406baf09ce33339e3c3f76e02097225a7240d6db7d1a247c194138d7
Sep 09 12:45:01.056330 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:01.056296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg" event={"ID":"1ef22df4-2239-4a4d-8adf-da77f3ae503f","Type":"ContainerStarted","Data":"5a220b41406baf09ce33339e3c3f76e02097225a7240d6db7d1a247c194138d7"}
Sep 09 12:45:01.057576 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:01.057551    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj" event={"ID":"31cef956-7266-4c66-a0ef-28366e6670ec","Type":"ContainerStarted","Data":"655feea374e58bf8ea113f16cf3cd1601f9a2dd7a01a8775db5056f669fde7a7"}
Sep 09 12:45:01.058633 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:01.058613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb" event={"ID":"bea9daa7-5a52-4dda-888f-d7256983c5cb","Type":"ContainerStarted","Data":"0b34aee17d290b892e0fdcc37ea29dbcb7604be2c9d71fbce0cd1a9faaecc30f"}
Sep 09 12:45:04.079548 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:04.079472    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ef22df4-2239-4a4d-8adf-da77f3ae503f" containerID="f147a903108a08e0ccd2772037afe4d05023c2cdeb283e64ebc9c6cb91feeb17" exitCode=0
Sep 09 12:45:04.079929 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:04.079536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg" event={"ID":"1ef22df4-2239-4a4d-8adf-da77f3ae503f","Type":"ContainerDied","Data":"f147a903108a08e0ccd2772037afe4d05023c2cdeb283e64ebc9c6cb91feeb17"}
Sep 09 12:45:04.081069 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:04.081051    2578 generic.go:356] "Generic (PLEG): container finished" podID="31cef956-7266-4c66-a0ef-28366e6670ec" containerID="cfcc0c5c479c82bf709afccb583a04462e74b3175cdf5163fc7917e4ef1c58bc" exitCode=0
Sep 09 12:45:04.081158 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:04.081109    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj" event={"ID":"31cef956-7266-4c66-a0ef-28366e6670ec","Type":"ContainerDied","Data":"cfcc0c5c479c82bf709afccb583a04462e74b3175cdf5163fc7917e4ef1c58bc"}
Sep 09 12:45:04.082609 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:04.082582    2578 generic.go:356] "Generic (PLEG): container finished" podID="bea9daa7-5a52-4dda-888f-d7256983c5cb" containerID="6f28bc1ffee887dde6657be66913102a42a0c2eabc6fe24930d5b8b91e4515f6" exitCode=0
Sep 09 12:45:04.082719 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:04.082664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb" event={"ID":"bea9daa7-5a52-4dda-888f-d7256983c5cb","Type":"ContainerDied","Data":"6f28bc1ffee887dde6657be66913102a42a0c2eabc6fe24930d5b8b91e4515f6"}
Sep 09 12:45:05.227628 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:05.227599    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj"
Sep 09 12:45:05.269488 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:05.269466    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg"
Sep 09 12:45:05.307402 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:05.307377    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb"
Sep 09 12:45:06.093659 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:06.093632    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj"
Sep 09 12:45:06.093834 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:06.093631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj" event={"ID":"31cef956-7266-4c66-a0ef-28366e6670ec","Type":"ContainerDied","Data":"655feea374e58bf8ea113f16cf3cd1601f9a2dd7a01a8775db5056f669fde7a7"}
Sep 09 12:45:06.093834 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:06.093794    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="655feea374e58bf8ea113f16cf3cd1601f9a2dd7a01a8775db5056f669fde7a7"
Sep 09 12:45:06.094976 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:06.094961    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb"
Sep 09 12:45:06.095094 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:06.095004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb" event={"ID":"bea9daa7-5a52-4dda-888f-d7256983c5cb","Type":"ContainerDied","Data":"0b34aee17d290b892e0fdcc37ea29dbcb7604be2c9d71fbce0cd1a9faaecc30f"}
Sep 09 12:45:06.095094 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:06.095040    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b34aee17d290b892e0fdcc37ea29dbcb7604be2c9d71fbce0cd1a9faaecc30f"
Sep 09 12:45:06.096516 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:06.096496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg" event={"ID":"1ef22df4-2239-4a4d-8adf-da77f3ae503f","Type":"ContainerDied","Data":"5a220b41406baf09ce33339e3c3f76e02097225a7240d6db7d1a247c194138d7"}
Sep 09 12:45:06.096619 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:06.096525    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg"
Sep 09 12:45:06.096619 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:06.096530    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a220b41406baf09ce33339e3c3f76e02097225a7240d6db7d1a247c194138d7"
Sep 09 12:45:08.736168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:08.736136    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg"]
Sep 09 12:45:08.738083 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:08.738056    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3f4a7d-29815965-h4rcg"]
Sep 09 12:45:08.774005 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:08.773975    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"]
Sep 09 12:45:08.774386 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:08.774277    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb" podUID="37b78815-223c-48af-94af-2d5a776065b5" containerName="payload-processing" containerID="cri-o://e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d" gracePeriod=30
Sep 09 12:45:09.039968 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.039947    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:45:09.113311 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.113280    2578 generic.go:356] "Generic (PLEG): container finished" podID="37b78815-223c-48af-94af-2d5a776065b5" containerID="e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d" exitCode=0
Sep 09 12:45:09.113462 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.113329    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"
Sep 09 12:45:09.113462 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.113376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb" event={"ID":"37b78815-223c-48af-94af-2d5a776065b5","Type":"ContainerDied","Data":"e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d"}
Sep 09 12:45:09.113462 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.113436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb" event={"ID":"37b78815-223c-48af-94af-2d5a776065b5","Type":"ContainerDied","Data":"9bb0d5cb920f51e3bf1f0cc0994dac2586d1bebf5e0d81ef4974d606fb1b7752"}
Sep 09 12:45:09.113462 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.113459    2578 scope.go:117] "RemoveContainer" containerID="e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d"
Sep 09 12:45:09.122912 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.122893    2578 scope.go:117] "RemoveContainer" containerID="e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d"
Sep 09 12:45:09.123191 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:45:09.123172    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d\": container with ID starting with e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d not found: ID does not exist" containerID="e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d"
Sep 09 12:45:09.123292 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.123199    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d"} err="failed to get container status \"e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d\": rpc error: code = NotFound desc = could not find container \"e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d\": container with ID starting with e52487557562c5e907b300cba57c6d8fc040e5f087f6850832ef77e43fa1ee9d not found: ID does not exist"
Sep 09 12:45:09.127588 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.127565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b78815-223c-48af-94af-2d5a776065b5-plugins-config-volume\") pod \"37b78815-223c-48af-94af-2d5a776065b5\" (UID: \"37b78815-223c-48af-94af-2d5a776065b5\") "
Sep 09 12:45:09.127655 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.127628    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9nlj\" (UniqueName: \"kubernetes.io/projected/37b78815-223c-48af-94af-2d5a776065b5-kube-api-access-w9nlj\") pod \"37b78815-223c-48af-94af-2d5a776065b5\" (UID: \"37b78815-223c-48af-94af-2d5a776065b5\") "
Sep 09 12:45:09.127974 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.127951    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b78815-223c-48af-94af-2d5a776065b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37b78815-223c-48af-94af-2d5a776065b5" (UID: "37b78815-223c-48af-94af-2d5a776065b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:45:09.129791 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.129763    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b78815-223c-48af-94af-2d5a776065b5-kube-api-access-w9nlj" (OuterVolumeSpecName: "kube-api-access-w9nlj") pod "37b78815-223c-48af-94af-2d5a776065b5" (UID: "37b78815-223c-48af-94af-2d5a776065b5"). InnerVolumeSpecName "kube-api-access-w9nlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:45:09.228616 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.228582    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b78815-223c-48af-94af-2d5a776065b5-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:45:09.228616 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.228608    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9nlj\" (UniqueName: \"kubernetes.io/projected/37b78815-223c-48af-94af-2d5a776065b5-kube-api-access-w9nlj\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:45:09.434495 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.434459    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"]
Sep 09 12:45:09.444196 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:09.440333    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3f4a7d-5ccb9f8d95-8hnxb"]
Sep 09 12:45:10.186301 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:10.186269    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ef22df4-2239-4a4d-8adf-da77f3ae503f" path="/var/lib/kubelet/pods/1ef22df4-2239-4a4d-8adf-da77f3ae503f/volumes"
Sep 09 12:45:10.186658 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:10.186606    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b78815-223c-48af-94af-2d5a776065b5" path="/var/lib/kubelet/pods/37b78815-223c-48af-94af-2d5a776065b5/volumes"
Sep 09 12:45:13.230524 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:13.230490    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:45:22.222823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.222783    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"]
Sep 09 12:45:22.223283 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223265    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cef956-7266-4c66-a0ef-28366e6670ec" containerName="cleanup"
Sep 09 12:45:22.223332 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223286    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cef956-7266-4c66-a0ef-28366e6670ec" containerName="cleanup"
Sep 09 12:45:22.223332 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223303    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea9daa7-5a52-4dda-888f-d7256983c5cb" containerName="cleanup"
Sep 09 12:45:22.223332 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223309    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea9daa7-5a52-4dda-888f-d7256983c5cb" containerName="cleanup"
Sep 09 12:45:22.223332 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223317    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef22df4-2239-4a4d-8adf-da77f3ae503f" containerName="cleanup"
Sep 09 12:45:22.223332 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223322    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef22df4-2239-4a4d-8adf-da77f3ae503f" containerName="cleanup"
Sep 09 12:45:22.223488 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223337    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b78815-223c-48af-94af-2d5a776065b5" containerName="payload-processing"
Sep 09 12:45:22.223488 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223342    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b78815-223c-48af-94af-2d5a776065b5" containerName="payload-processing"
Sep 09 12:45:22.223488 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223414    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bea9daa7-5a52-4dda-888f-d7256983c5cb" containerName="cleanup"
Sep 09 12:45:22.223488 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223422    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b78815-223c-48af-94af-2d5a776065b5" containerName="payload-processing"
Sep 09 12:45:22.223488 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223434    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef22df4-2239-4a4d-8adf-da77f3ae503f" containerName="cleanup"
Sep 09 12:45:22.223488 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.223444    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="31cef956-7266-4c66-a0ef-28366e6670ec" containerName="cleanup"
Sep 09 12:45:22.227690 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.227674    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.230433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.230403    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fe6606\"/\"kube-root-ca.crt\""
Sep 09 12:45:22.230575 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.230408    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fe6606\"/\"default-dockercfg-vxvmq\""
Sep 09 12:45:22.230575 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.230414    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fe6606\"/\"rate-test-model-fe6606-kserve-self-signed-certs\""
Sep 09 12:45:22.230575 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.230469    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fe6606\"/\"openshift-service-ca.crt\""
Sep 09 12:45:22.234801 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.234777    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"]
Sep 09 12:45:22.266753 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.266725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tmp-dir\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.266882 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.266780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-model-cache\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.266882 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.266818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-home\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.266882 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.266847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tls-certs\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.267048 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.266907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzckm\" (UniqueName: \"kubernetes.io/projected/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-kube-api-access-wzckm\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.267048 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.266990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-dshm\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.368150 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.368108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-home\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.368150 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.368151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tls-certs\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.368434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.368188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzckm\" (UniqueName: \"kubernetes.io/projected/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-kube-api-access-wzckm\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.368434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.368253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-dshm\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.368434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.368301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tmp-dir\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.368434 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.368358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-model-cache\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.368657 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.368611    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-home\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.368657 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.368648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tmp-dir\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.368756 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.368737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-model-cache\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.370675 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.370651    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-dshm\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.370985 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.370965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tls-certs\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.376145 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.376119    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzckm\" (UniqueName: \"kubernetes.io/projected/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-kube-api-access-wzckm\") pod \"rate-test-model-fe6606-kserve-d89765f75-j8284\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") " pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.538008 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.537910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:22.809787 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:22.809758    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"]
Sep 09 12:45:22.811806 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:45:22.811778    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod404d8d8f_575a_4fd0_bc81_649bfcdd3ae5.slice/crio-7be93fd144e4b2d282dea42c1b7932846744ffc906d102a9693990727e4ba09f WatchSource:0}: Error finding container 7be93fd144e4b2d282dea42c1b7932846744ffc906d102a9693990727e4ba09f: Status 404 returned error can't find the container with id 7be93fd144e4b2d282dea42c1b7932846744ffc906d102a9693990727e4ba09f
Sep 09 12:45:23.189861 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:23.189830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284" event={"ID":"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5","Type":"ContainerStarted","Data":"bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277"}
Sep 09 12:45:23.190014 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:23.189868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284" event={"ID":"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5","Type":"ContainerStarted","Data":"7be93fd144e4b2d282dea42c1b7932846744ffc906d102a9693990727e4ba09f"}
Sep 09 12:45:23.214591 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:23.214538    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284" podStartSLOduration=1.214521373 podStartE2EDuration="1.214521373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:45:23.213005314 +0000 UTC m=+1543.556603261" watchObservedRunningTime="2026-09-09 12:45:23.214521373 +0000 UTC m=+1543.558119320"
Sep 09 12:45:26.876636 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:26.876599    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb"]
Sep 09 12:45:26.878476 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:26.878457    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a3f43f-29815965-s27mb"]
Sep 09 12:45:26.939414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:26.939381    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"]
Sep 09 12:45:26.939617 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:26.939595    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d" podUID="c92ca333-f4d8-4d40-a6e9-33cefcccb2b2" containerName="payload-pre-processing" containerID="cri-o://e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b" gracePeriod=30
Sep 09 12:45:27.003847 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.003809    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d" podUID="c92ca333-f4d8-4d40-a6e9-33cefcccb2b2" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.59:9004: connect: connection refused"
Sep 09 12:45:27.191639 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.191614    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:45:27.209684 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.209660    2578 generic.go:356] "Generic (PLEG): container finished" podID="c92ca333-f4d8-4d40-a6e9-33cefcccb2b2" containerID="e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b" exitCode=0
Sep 09 12:45:27.209821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.209703    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"
Sep 09 12:45:27.209821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.209746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d" event={"ID":"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2","Type":"ContainerDied","Data":"e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b"}
Sep 09 12:45:27.209821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.209797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d" event={"ID":"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2","Type":"ContainerDied","Data":"f2f652010c88da48b6f5f4e7e3d229a3530bf94a3606b498fd9987b8eecbcd9e"}
Sep 09 12:45:27.209821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.209820    2578 scope.go:117] "RemoveContainer" containerID="e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b"
Sep 09 12:45:27.222374 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.222342    2578 scope.go:117] "RemoveContainer" containerID="e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b"
Sep 09 12:45:27.222848 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:45:27.222825    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b\": container with ID starting with e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b not found: ID does not exist" containerID="e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b"
Sep 09 12:45:27.222943 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.222857    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b"} err="failed to get container status \"e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b\": rpc error: code = NotFound desc = could not find container \"e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b\": container with ID starting with e7f2bd240f15cf23854e8e40d2a0cdf88c5aff7de00dbba2a03444b72133367b not found: ID does not exist"
Sep 09 12:45:27.316188 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.316155    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-plugins-config-volume\") pod \"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2\" (UID: \"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2\") "
Sep 09 12:45:27.316376 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.316245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwgjd\" (UniqueName: \"kubernetes.io/projected/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-kube-api-access-nwgjd\") pod \"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2\" (UID: \"c92ca333-f4d8-4d40-a6e9-33cefcccb2b2\") "
Sep 09 12:45:27.316581 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.316560    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c92ca333-f4d8-4d40-a6e9-33cefcccb2b2" (UID: "c92ca333-f4d8-4d40-a6e9-33cefcccb2b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:45:27.318479 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.318452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-kube-api-access-nwgjd" (OuterVolumeSpecName: "kube-api-access-nwgjd") pod "c92ca333-f4d8-4d40-a6e9-33cefcccb2b2" (UID: "c92ca333-f4d8-4d40-a6e9-33cefcccb2b2"). InnerVolumeSpecName "kube-api-access-nwgjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:45:27.417909 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.417877    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:45:27.417909 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.417906    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwgjd\" (UniqueName: \"kubernetes.io/projected/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2-kube-api-access-nwgjd\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:45:27.431139 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.431056    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:45:27.531201 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.531170    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"]
Sep 09 12:45:27.534981 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:27.534962    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a3f43f-699b75f7b6-kmr4d"]
Sep 09 12:45:28.186608 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:28.186572    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bea9daa7-5a52-4dda-888f-d7256983c5cb" path="/var/lib/kubelet/pods/bea9daa7-5a52-4dda-888f-d7256983c5cb/volumes"
Sep 09 12:45:28.187172 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:28.187152    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c92ca333-f4d8-4d40-a6e9-33cefcccb2b2" path="/var/lib/kubelet/pods/c92ca333-f4d8-4d40-a6e9-33cefcccb2b2/volumes"
Sep 09 12:45:32.538108 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:32.538075    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:32.540435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:32.538136    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:32.550559 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:32.550539    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:33.254052 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:33.254027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:45:45.326301 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.326271    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj"]
Sep 09 12:45:45.329359 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.329333    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-590823-29815965-w7xfj"]
Sep 09 12:45:45.371440 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.371408    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"]
Sep 09 12:45:45.371690 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.371649    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj" podUID="3c8e5a13-9f30-4c93-b9ea-e24c1e54a641" containerName="payload-pre-processing" containerID="cri-o://1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8" gracePeriod=30
Sep 09 12:45:45.537955 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.537922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:45:45.639627 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.639595    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:45:45.807302 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.807266    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-plugins-config-volume\") pod \"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641\" (UID: \"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641\") "
Sep 09 12:45:45.807515 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.807379    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpzd8\" (UniqueName: \"kubernetes.io/projected/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-kube-api-access-zpzd8\") pod \"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641\" (UID: \"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641\") "
Sep 09 12:45:45.807718 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.807694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c8e5a13-9f30-4c93-b9ea-e24c1e54a641" (UID: "3c8e5a13-9f30-4c93-b9ea-e24c1e54a641"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:45:45.809653 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.809629    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-kube-api-access-zpzd8" (OuterVolumeSpecName: "kube-api-access-zpzd8") pod "3c8e5a13-9f30-4c93-b9ea-e24c1e54a641" (UID: "3c8e5a13-9f30-4c93-b9ea-e24c1e54a641"). InnerVolumeSpecName "kube-api-access-zpzd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:45:45.908566 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.908538    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:45:45.908566 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:45.908564    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpzd8\" (UniqueName: \"kubernetes.io/projected/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641-kube-api-access-zpzd8\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:45:46.185932 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:46.185857    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31cef956-7266-4c66-a0ef-28366e6670ec" path="/var/lib/kubelet/pods/31cef956-7266-4c66-a0ef-28366e6670ec/volumes"
Sep 09 12:45:46.312665 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:46.312626    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c8e5a13-9f30-4c93-b9ea-e24c1e54a641" containerID="1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8" exitCode=0
Sep 09 12:45:46.312864 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:46.312682    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"
Sep 09 12:45:46.312864 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:46.312713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj" event={"ID":"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641","Type":"ContainerDied","Data":"1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8"}
Sep 09 12:45:46.312864 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:46.312775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj" event={"ID":"3c8e5a13-9f30-4c93-b9ea-e24c1e54a641","Type":"ContainerDied","Data":"a6b7d4e5a7ff1bd3c9b45ce57c14b6cde8278050a336ba86424a1f96d83d76e7"}
Sep 09 12:45:46.312864 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:46.312802    2578 scope.go:117] "RemoveContainer" containerID="1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8"
Sep 09 12:45:46.321878 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:46.321862    2578 scope.go:117] "RemoveContainer" containerID="1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8"
Sep 09 12:45:46.322131 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:45:46.322115    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8\": container with ID starting with 1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8 not found: ID does not exist" containerID="1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8"
Sep 09 12:45:46.322180 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:46.322139    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8"} err="failed to get container status \"1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8\": rpc error: code = NotFound desc = could not find container \"1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8\": container with ID starting with 1274df134834236025b1e81022fb4e4d971c9b653b5544ac97534ead5d9932e8 not found: ID does not exist"
Sep 09 12:45:46.328059 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:46.328039    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"]
Sep 09 12:45:46.331995 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:46.331974    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-590823-8bd96d5f8-8rtmj"]
Sep 09 12:45:48.187079 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:48.187049    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c8e5a13-9f30-4c93-b9ea-e24c1e54a641" path="/var/lib/kubelet/pods/3c8e5a13-9f30-4c93-b9ea-e24c1e54a641/volumes"
Sep 09 12:45:55.532750 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:45:55.532714    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:46:03.455621 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:03.455589    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"]
Sep 09 12:46:05.828100 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:05.828060    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:46:18.475898 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.475828    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284" podUID="404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" containerName="main" containerID="cri-o://bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277" gracePeriod=45
Sep 09 12:46:18.736991 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.736922    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:46:18.828620 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.828589    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-dshm\") pod \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") "
Sep 09 12:46:18.828821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.828656    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-home\") pod \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") "
Sep 09 12:46:18.828821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.828689    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tmp-dir\") pod \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") "
Sep 09 12:46:18.828821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.828714    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzckm\" (UniqueName: \"kubernetes.io/projected/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-kube-api-access-wzckm\") pod \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") "
Sep 09 12:46:18.828821 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.828767    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tls-certs\") pod \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") "
Sep 09 12:46:18.829038 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.828855    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-model-cache\") pod \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\" (UID: \"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5\") "
Sep 09 12:46:18.829038 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.828999    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" (UID: "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:46:18.829038 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.829014    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-home" (OuterVolumeSpecName: "home") pod "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" (UID: "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:46:18.829169 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.829138    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-model-cache" (OuterVolumeSpecName: "model-cache") pod "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" (UID: "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:46:18.829370 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.829351    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-home\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:46:18.829447 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.829377    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tmp-dir\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:46:18.829447 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.829390    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-model-cache\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:46:18.830914 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.830893    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-dshm" (OuterVolumeSpecName: "dshm") pod "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" (UID: "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:46:18.831013 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.830996    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" (UID: "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:46:18.831057 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.831032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-kube-api-access-wzckm" (OuterVolumeSpecName: "kube-api-access-wzckm") pod "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" (UID: "404d8d8f-575a-4fd0-bc81-649bfcdd3ae5"). InnerVolumeSpecName "kube-api-access-wzckm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:46:18.930737 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.930697    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-dshm\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:46:18.930737 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.930730    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzckm\" (UniqueName: \"kubernetes.io/projected/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-kube-api-access-wzckm\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:46:18.930737 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:18.930742    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5-tls-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:46:19.492339 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:19.492305    2578 generic.go:356] "Generic (PLEG): container finished" podID="404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" containerID="bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277" exitCode=0
Sep 09 12:46:19.492729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:19.492351    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"
Sep 09 12:46:19.492729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:19.492395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284" event={"ID":"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5","Type":"ContainerDied","Data":"bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277"}
Sep 09 12:46:19.492729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:19.492443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284" event={"ID":"404d8d8f-575a-4fd0-bc81-649bfcdd3ae5","Type":"ContainerDied","Data":"7be93fd144e4b2d282dea42c1b7932846744ffc906d102a9693990727e4ba09f"}
Sep 09 12:46:19.492729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:19.492466    2578 scope.go:117] "RemoveContainer" containerID="bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277"
Sep 09 12:46:19.501732 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:19.501714    2578 scope.go:117] "RemoveContainer" containerID="bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277"
Sep 09 12:46:19.501974 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:46:19.501959    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277\": container with ID starting with bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277 not found: ID does not exist" containerID="bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277"
Sep 09 12:46:19.502028 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:19.501982    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277"} err="failed to get container status \"bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277\": rpc error: code = NotFound desc = could not find container \"bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277\": container with ID starting with bd0f77c4a2c332e6d59a312585a488efe7a3eb7b660c2553d3fcc637e6c03277 not found: ID does not exist"
Sep 09 12:46:19.513436 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:19.513412    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"]
Sep 09 12:46:19.519441 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:19.519418    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fe6606/rate-test-model-fe6606-kserve-d89765f75-j8284"]
Sep 09 12:46:19.725813 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:19.725781    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:46:20.186100 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:20.186071    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" path="/var/lib/kubelet/pods/404d8d8f-575a-4fd0-bc81-649bfcdd3ae5/volumes"
Sep 09 12:46:37.527625 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.527547    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"]
Sep 09 12:46:37.528135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.527966    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8e5a13-9f30-4c93-b9ea-e24c1e54a641" containerName="payload-pre-processing"
Sep 09 12:46:37.528135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.527979    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8e5a13-9f30-4c93-b9ea-e24c1e54a641" containerName="payload-pre-processing"
Sep 09 12:46:37.528135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.527995    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" containerName="main"
Sep 09 12:46:37.528135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.528004    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" containerName="main"
Sep 09 12:46:37.528135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.528018    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c92ca333-f4d8-4d40-a6e9-33cefcccb2b2" containerName="payload-pre-processing"
Sep 09 12:46:37.528135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.528025    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c92ca333-f4d8-4d40-a6e9-33cefcccb2b2" containerName="payload-pre-processing"
Sep 09 12:46:37.528135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.528095    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="404d8d8f-575a-4fd0-bc81-649bfcdd3ae5" containerName="main"
Sep 09 12:46:37.528135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.528106    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c92ca333-f4d8-4d40-a6e9-33cefcccb2b2" containerName="payload-pre-processing"
Sep 09 12:46:37.528135 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.528116    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c8e5a13-9f30-4c93-b9ea-e24c1e54a641" containerName="payload-pre-processing"
Sep 09 12:46:37.532501 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.532483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:37.534483 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.534464    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2db0a6-serving-cert\""
Sep 09 12:46:37.541426 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.541402    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"]
Sep 09 12:46:37.610252 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.610198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:37.610430 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.610353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:37.610493 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.610445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-metrics-certs\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:37.610611 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.610569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmlmx\" (UniqueName: \"kubernetes.io/projected/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-kube-api-access-lmlmx\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:37.711599 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.711567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-metrics-certs\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:37.711792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.711649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlmx\" (UniqueName: \"kubernetes.io/projected/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-kube-api-access-lmlmx\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:37.711792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.711681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:37.711792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.711724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:37.711939 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:46:37.711808    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-serving-cert: secret "maas-api-e2e-ipp-b-2db0a6-serving-cert" not found
Sep 09 12:46:37.711939 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:46:37.711873    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-maas-api-tls podName:ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21 nodeName:}" failed. No retries permitted until 2026-09-09 12:46:38.211856596 +0000 UTC m=+1618.555454546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-maas-api-tls") pod "maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh" (UID: "ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21") : secret "maas-api-e2e-ipp-b-2db0a6-serving-cert" not found
Sep 09 12:46:37.712382 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.712364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:37.714107 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.714080    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-metrics-certs\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:37.720259 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:37.720203    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlmx\" (UniqueName: \"kubernetes.io/projected/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-kube-api-access-lmlmx\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:38.217771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.217735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:38.220355 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.220327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:38.249465 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.249434    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"]
Sep 09 12:46:38.253187 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.253167    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:46:38.255478 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.255459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2db0a6-dockercfg-ppf4p\""
Sep 09 12:46:38.255601 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.255496    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2db0a6\""
Sep 09 12:46:38.263289 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.263263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"]
Sep 09 12:46:38.318352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.318308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdfw\" (UniqueName: \"kubernetes.io/projected/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-kube-api-access-vjdfw\") pod \"payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7\" (UID: \"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:46:38.318533 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.318428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7\" (UID: \"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:46:38.418973 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.418942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdfw\" (UniqueName: \"kubernetes.io/projected/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-kube-api-access-vjdfw\") pod \"payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7\" (UID: \"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:46:38.419157 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.418998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7\" (UID: \"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:46:38.419666 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.419646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7\" (UID: \"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:46:38.426741 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.426721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdfw\" (UniqueName: \"kubernetes.io/projected/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-kube-api-access-vjdfw\") pod \"payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7\" (UID: \"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:46:38.444605 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.444579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:38.565799 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.565752    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:46:38.714201 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.714177    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"]
Sep 09 12:46:38.716293 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:46:38.716262    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5dd6e7_ad9c_4f0d_b5f4_59bd0b6a1c21.slice/crio-1f562cbc996fd4e12cffcfd02e630956f212aa9a39ccdaad53b857a43e57bd9f WatchSource:0}: Error finding container 1f562cbc996fd4e12cffcfd02e630956f212aa9a39ccdaad53b857a43e57bd9f: Status 404 returned error can't find the container with id 1f562cbc996fd4e12cffcfd02e630956f212aa9a39ccdaad53b857a43e57bd9f
Sep 09 12:46:38.842439 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:38.842411    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"]
Sep 09 12:46:38.844880 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:46:38.844851    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd6b3fe_674b_4ad7_8fe4_decd7009db3a.slice/crio-9e31c1b22258afe0448fc40c2278de7f3893b9c10737f857e06d91f65ec2b593 WatchSource:0}: Error finding container 9e31c1b22258afe0448fc40c2278de7f3893b9c10737f857e06d91f65ec2b593: Status 404 returned error can't find the container with id 9e31c1b22258afe0448fc40c2278de7f3893b9c10737f857e06d91f65ec2b593
Sep 09 12:46:39.599024 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:39.598986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7" event={"ID":"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a","Type":"ContainerStarted","Data":"33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806"}
Sep 09 12:46:39.599474 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:39.599048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7" event={"ID":"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a","Type":"ContainerStarted","Data":"9e31c1b22258afe0448fc40c2278de7f3893b9c10737f857e06d91f65ec2b593"}
Sep 09 12:46:39.599474 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:39.599111    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:46:39.600502 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:39.600480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh" event={"ID":"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21","Type":"ContainerStarted","Data":"c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0"}
Sep 09 12:46:39.600568 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:39.600524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh" event={"ID":"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21","Type":"ContainerStarted","Data":"1f562cbc996fd4e12cffcfd02e630956f212aa9a39ccdaad53b857a43e57bd9f"}
Sep 09 12:46:39.600568 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:39.600560    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:39.616233 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:39.616176    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7" podStartSLOduration=1.616160356 podStartE2EDuration="1.616160356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:46:39.613443148 +0000 UTC m=+1619.957041097" watchObservedRunningTime="2026-09-09 12:46:39.616160356 +0000 UTC m=+1619.959758305"
Sep 09 12:46:39.630319 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:39.630271    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh" podStartSLOduration=2.318318202 podStartE2EDuration="2.630254628s" totalImagesPullingTime="311.936426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:46:39.628277208 +0000 UTC m=+1619.971875349" watchObservedRunningTime="2026-09-09 12:46:39.630254628 +0000 UTC m=+1619.973852575"
Sep 09 12:46:40.229405 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:40.229355    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:46:45.611576 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:45.611545    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:46:50.607082 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:46:50.607051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:47:15.036452 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:15.036413    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:47:27.931833 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:27.931790    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:47:39.822376 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:39.822338    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"]
Sep 09 12:47:39.826767 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:39.826743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:39.828629 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:39.828608    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-90dee1-serving-cert\""
Sep 09 12:47:39.838621 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:39.838597    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"]
Sep 09 12:47:39.932106 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:39.932075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:39.932310 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:39.932118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:39.932310 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:39.932238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxpld\" (UniqueName: \"kubernetes.io/projected/67bfbe6f-43c9-48fd-b871-8d9670660502-kube-api-access-cxpld\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:39.932310 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:39.932288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bfbe6f-43c9-48fd-b871-8d9670660502-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:40.033536 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.033506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:40.033724 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.033583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxpld\" (UniqueName: \"kubernetes.io/projected/67bfbe6f-43c9-48fd-b871-8d9670660502-kube-api-access-cxpld\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:40.033724 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.033628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bfbe6f-43c9-48fd-b871-8d9670660502-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:40.033724 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.033674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:40.034370 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.034342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bfbe6f-43c9-48fd-b871-8d9670660502-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:40.036112 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.036089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:40.036232 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.036144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:40.041189 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.041169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxpld\" (UniqueName: \"kubernetes.io/projected/67bfbe6f-43c9-48fd-b871-8d9670660502-kube-api-access-cxpld\") pod \"maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:40.138884 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.138785    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:40.413827 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.413793    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"]
Sep 09 12:47:40.452097 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.452062    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"]
Sep 09 12:47:40.456745 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.456730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:47:40.458691 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.458674    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-90dee1\""
Sep 09 12:47:40.458753 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.458743    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-90dee1-dockercfg-92mxn\""
Sep 09 12:47:40.464481 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.464458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"]
Sep 09 12:47:40.537877 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.537837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf20340-1e39-430b-adea-a224f7557aaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c\" (UID: \"bbf20340-1e39-430b-adea-a224f7557aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:47:40.538052 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.537889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzj9n\" (UniqueName: \"kubernetes.io/projected/bbf20340-1e39-430b-adea-a224f7557aaf-kube-api-access-rzj9n\") pod \"payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c\" (UID: \"bbf20340-1e39-430b-adea-a224f7557aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:47:40.639507 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.639459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf20340-1e39-430b-adea-a224f7557aaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c\" (UID: \"bbf20340-1e39-430b-adea-a224f7557aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:47:40.639687 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.639525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzj9n\" (UniqueName: \"kubernetes.io/projected/bbf20340-1e39-430b-adea-a224f7557aaf-kube-api-access-rzj9n\") pod \"payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c\" (UID: \"bbf20340-1e39-430b-adea-a224f7557aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:47:40.640153 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.640134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf20340-1e39-430b-adea-a224f7557aaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c\" (UID: \"bbf20340-1e39-430b-adea-a224f7557aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:47:40.647404 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.647383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzj9n\" (UniqueName: \"kubernetes.io/projected/bbf20340-1e39-430b-adea-a224f7557aaf-kube-api-access-rzj9n\") pod \"payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c\" (UID: \"bbf20340-1e39-430b-adea-a224f7557aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:47:40.769243 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.769127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:47:40.923631 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:40.923595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq" event={"ID":"67bfbe6f-43c9-48fd-b871-8d9670660502","Type":"ContainerStarted","Data":"62534d85ef7f12d2f29303aa77198ff4b5fbcb0e5260e45c344bb5cef3394ec1"}
Sep 09 12:47:41.042607 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:41.042529    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"]
Sep 09 12:47:41.045297 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:47:41.045273    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf20340_1e39_430b_adea_a224f7557aaf.slice/crio-365e9e23c8805b6ef18dde2e901c131dc3ff095cecf0d04ec7425ed838dd39a3 WatchSource:0}: Error finding container 365e9e23c8805b6ef18dde2e901c131dc3ff095cecf0d04ec7425ed838dd39a3: Status 404 returned error can't find the container with id 365e9e23c8805b6ef18dde2e901c131dc3ff095cecf0d04ec7425ed838dd39a3
Sep 09 12:47:41.930130 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:41.930094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq" event={"ID":"67bfbe6f-43c9-48fd-b871-8d9670660502","Type":"ContainerStarted","Data":"6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7"}
Sep 09 12:47:41.930611 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:41.930191    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:41.931758 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:41.931729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c" event={"ID":"bbf20340-1e39-430b-adea-a224f7557aaf","Type":"ContainerStarted","Data":"37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39"}
Sep 09 12:47:41.931877 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:41.931776    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:47:41.931877 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:41.931793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c" event={"ID":"bbf20340-1e39-430b-adea-a224f7557aaf","Type":"ContainerStarted","Data":"365e9e23c8805b6ef18dde2e901c131dc3ff095cecf0d04ec7425ed838dd39a3"}
Sep 09 12:47:41.947015 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:41.946968    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq" podStartSLOduration=2.390628527 podStartE2EDuration="2.946954538s" totalImagesPullingTime="556.326011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:47:41.945003752 +0000 UTC m=+1682.288601714" watchObservedRunningTime="2026-09-09 12:47:41.946954538 +0000 UTC m=+1682.290552486"
Sep 09 12:47:41.959792 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:41.959740    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c" podStartSLOduration=1.959721498 podStartE2EDuration="1.959721498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:47:41.958872775 +0000 UTC m=+1682.302470715" watchObservedRunningTime="2026-09-09 12:47:41.959721498 +0000 UTC m=+1682.303319447"
Sep 09 12:47:46.936389 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:46.936349    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:47:47.942376 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:47.942351    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:47:52.939835 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:47:52.939802    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:48:10.329643 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.329544    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"]
Sep 09 12:48:10.330136 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.329771    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq" podUID="67bfbe6f-43c9-48fd-b871-8d9670660502" containerName="maas-api" containerID="cri-o://6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7" gracePeriod=30
Sep 09 12:48:10.399352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.399313    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"]
Sep 09 12:48:10.400280 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.399568    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c" podUID="bbf20340-1e39-430b-adea-a224f7557aaf" containerName="payload-pre-processing" containerID="cri-o://37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39" gracePeriod=30
Sep 09 12:48:10.665613 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.665583    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:48:10.690849 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.690822    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:48:10.832749 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.832720    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-maas-api-tls\") pod \"67bfbe6f-43c9-48fd-b871-8d9670660502\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") "
Sep 09 12:48:10.832935 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.832778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzj9n\" (UniqueName: \"kubernetes.io/projected/bbf20340-1e39-430b-adea-a224f7557aaf-kube-api-access-rzj9n\") pod \"bbf20340-1e39-430b-adea-a224f7557aaf\" (UID: \"bbf20340-1e39-430b-adea-a224f7557aaf\") "
Sep 09 12:48:10.832935 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.832834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxpld\" (UniqueName: \"kubernetes.io/projected/67bfbe6f-43c9-48fd-b871-8d9670660502-kube-api-access-cxpld\") pod \"67bfbe6f-43c9-48fd-b871-8d9670660502\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") "
Sep 09 12:48:10.832935 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.832857    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bfbe6f-43c9-48fd-b871-8d9670660502-openshift-service-ca\") pod \"67bfbe6f-43c9-48fd-b871-8d9670660502\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") "
Sep 09 12:48:10.832935 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.832895    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-metrics-certs\") pod \"67bfbe6f-43c9-48fd-b871-8d9670660502\" (UID: \"67bfbe6f-43c9-48fd-b871-8d9670660502\") "
Sep 09 12:48:10.832935 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.832936    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf20340-1e39-430b-adea-a224f7557aaf-plugins-config-volume\") pod \"bbf20340-1e39-430b-adea-a224f7557aaf\" (UID: \"bbf20340-1e39-430b-adea-a224f7557aaf\") "
Sep 09 12:48:10.833345 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.833312    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67bfbe6f-43c9-48fd-b871-8d9670660502-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67bfbe6f-43c9-48fd-b871-8d9670660502" (UID: "67bfbe6f-43c9-48fd-b871-8d9670660502"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:48:10.833560 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.833532    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf20340-1e39-430b-adea-a224f7557aaf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbf20340-1e39-430b-adea-a224f7557aaf" (UID: "bbf20340-1e39-430b-adea-a224f7557aaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:48:10.835204 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.835178    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "67bfbe6f-43c9-48fd-b871-8d9670660502" (UID: "67bfbe6f-43c9-48fd-b871-8d9670660502"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:48:10.835331 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.835232    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bfbe6f-43c9-48fd-b871-8d9670660502-kube-api-access-cxpld" (OuterVolumeSpecName: "kube-api-access-cxpld") pod "67bfbe6f-43c9-48fd-b871-8d9670660502" (UID: "67bfbe6f-43c9-48fd-b871-8d9670660502"). InnerVolumeSpecName "kube-api-access-cxpld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:48:10.835619 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.835600    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67bfbe6f-43c9-48fd-b871-8d9670660502" (UID: "67bfbe6f-43c9-48fd-b871-8d9670660502"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:48:10.835662 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.835604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf20340-1e39-430b-adea-a224f7557aaf-kube-api-access-rzj9n" (OuterVolumeSpecName: "kube-api-access-rzj9n") pod "bbf20340-1e39-430b-adea-a224f7557aaf" (UID: "bbf20340-1e39-430b-adea-a224f7557aaf"). InnerVolumeSpecName "kube-api-access-rzj9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:48:10.934170 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.934139    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxpld\" (UniqueName: \"kubernetes.io/projected/67bfbe6f-43c9-48fd-b871-8d9670660502-kube-api-access-cxpld\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:48:10.934170 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.934167    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bfbe6f-43c9-48fd-b871-8d9670660502-openshift-service-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:48:10.934387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.934177    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-metrics-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:48:10.934387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.934186    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf20340-1e39-430b-adea-a224f7557aaf-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:48:10.934387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.934194    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67bfbe6f-43c9-48fd-b871-8d9670660502-maas-api-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:48:10.934387 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:10.934202    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzj9n\" (UniqueName: \"kubernetes.io/projected/bbf20340-1e39-430b-adea-a224f7557aaf-kube-api-access-rzj9n\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:48:11.091700 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.091670    2578 generic.go:356] "Generic (PLEG): container finished" podID="67bfbe6f-43c9-48fd-b871-8d9670660502" containerID="6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7" exitCode=0
Sep 09 12:48:11.091879 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.091714    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"
Sep 09 12:48:11.091879 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.091758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq" event={"ID":"67bfbe6f-43c9-48fd-b871-8d9670660502","Type":"ContainerDied","Data":"6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7"}
Sep 09 12:48:11.091879 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.091863    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq" event={"ID":"67bfbe6f-43c9-48fd-b871-8d9670660502","Type":"ContainerDied","Data":"62534d85ef7f12d2f29303aa77198ff4b5fbcb0e5260e45c344bb5cef3394ec1"}
Sep 09 12:48:11.092042 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.091898    2578 scope.go:117] "RemoveContainer" containerID="6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7"
Sep 09 12:48:11.093183 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.093159    2578 generic.go:356] "Generic (PLEG): container finished" podID="bbf20340-1e39-430b-adea-a224f7557aaf" containerID="37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39" exitCode=0
Sep 09 12:48:11.093306 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.093196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c" event={"ID":"bbf20340-1e39-430b-adea-a224f7557aaf","Type":"ContainerDied","Data":"37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39"}
Sep 09 12:48:11.093306 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.093237    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"
Sep 09 12:48:11.093306 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.093262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c" event={"ID":"bbf20340-1e39-430b-adea-a224f7557aaf","Type":"ContainerDied","Data":"365e9e23c8805b6ef18dde2e901c131dc3ff095cecf0d04ec7425ed838dd39a3"}
Sep 09 12:48:11.102378 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.102356    2578 scope.go:117] "RemoveContainer" containerID="6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7"
Sep 09 12:48:11.102761 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:48:11.102657    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7\": container with ID starting with 6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7 not found: ID does not exist" containerID="6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7"
Sep 09 12:48:11.102761 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.102684    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7"} err="failed to get container status \"6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7\": rpc error: code = NotFound desc = could not find container \"6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7\": container with ID starting with 6f2baae1bd02c5427ab71d2591f1160dcd41bd40bd5fd54416a454673ef9dbe7 not found: ID does not exist"
Sep 09 12:48:11.102761 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.102700    2578 scope.go:117] "RemoveContainer" containerID="37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39"
Sep 09 12:48:11.111397 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.111379    2578 scope.go:117] "RemoveContainer" containerID="37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39"
Sep 09 12:48:11.111700 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:48:11.111681    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39\": container with ID starting with 37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39 not found: ID does not exist" containerID="37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39"
Sep 09 12:48:11.111749 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.111707    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39"} err="failed to get container status \"37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39\": rpc error: code = NotFound desc = could not find container \"37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39\": container with ID starting with 37b16f07de0dabd47a0e74f74f58be2acb48f238b869ba5a21b2f96fd0a08f39 not found: ID does not exist"
Sep 09 12:48:11.114966 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.114943    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"]
Sep 09 12:48:11.118752 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.118718    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-90dee1-64657d4668-7npjq"]
Sep 09 12:48:11.128548 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.128525    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"]
Sep 09 12:48:11.132115 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:11.132096    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-90dee1-5b78d889f8-sbw6c"]
Sep 09 12:48:12.186348 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:12.186316    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67bfbe6f-43c9-48fd-b871-8d9670660502" path="/var/lib/kubelet/pods/67bfbe6f-43c9-48fd-b871-8d9670660502/volumes"
Sep 09 12:48:12.186733 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:12.186692    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbf20340-1e39-430b-adea-a224f7557aaf" path="/var/lib/kubelet/pods/bbf20340-1e39-430b-adea-a224f7557aaf/volumes"
Sep 09 12:48:14.830834 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:14.830791    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:48:34.333616 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:34.333582    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:48:47.030526 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:48:47.030488    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:49:37.624436 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.624342    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"]
Sep 09 12:49:37.624938 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.624595    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh" podUID="ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21" containerName="maas-api" containerID="cri-o://c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0" gracePeriod=30
Sep 09 12:49:37.688644 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.688608    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"]
Sep 09 12:49:37.688971 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.688912    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7" podUID="dfd6b3fe-674b-4ad7-8fe4-decd7009db3a" containerName="payload-pre-processing" containerID="cri-o://33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806" gracePeriod=30
Sep 09 12:49:37.936104 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.936080    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:49:37.964401 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.964376    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:49:37.982351 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.982321    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-maas-api-tls\") pod \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") "
Sep 09 12:49:37.982533 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.982358    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-openshift-service-ca\") pod \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") "
Sep 09 12:49:37.982533 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.982378    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmlmx\" (UniqueName: \"kubernetes.io/projected/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-kube-api-access-lmlmx\") pod \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") "
Sep 09 12:49:37.982533 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.982514    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-metrics-certs\") pod \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\" (UID: \"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21\") "
Sep 09 12:49:37.982783 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.982749    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21" (UID: "ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:49:37.984725 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.984688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21" (UID: "ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:49:37.984725 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.984715    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21" (UID: "ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:49:37.984857 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:37.984737    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-kube-api-access-lmlmx" (OuterVolumeSpecName: "kube-api-access-lmlmx") pod "ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21" (UID: "ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21"). InnerVolumeSpecName "kube-api-access-lmlmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:49:38.083316 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.083276    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjdfw\" (UniqueName: \"kubernetes.io/projected/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-kube-api-access-vjdfw\") pod \"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a\" (UID: \"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a\") "
Sep 09 12:49:38.083519 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.083337    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-plugins-config-volume\") pod \"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a\" (UID: \"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a\") "
Sep 09 12:49:38.083643 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.083625    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-metrics-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:49:38.083703 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.083648    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-maas-api-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:49:38.083703 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.083660    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-openshift-service-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:49:38.083703 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.083673    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmlmx\" (UniqueName: \"kubernetes.io/projected/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21-kube-api-access-lmlmx\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:49:38.083808 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.083731    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dfd6b3fe-674b-4ad7-8fe4-decd7009db3a" (UID: "dfd6b3fe-674b-4ad7-8fe4-decd7009db3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:49:38.085505 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.085484    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-kube-api-access-vjdfw" (OuterVolumeSpecName: "kube-api-access-vjdfw") pod "dfd6b3fe-674b-4ad7-8fe4-decd7009db3a" (UID: "dfd6b3fe-674b-4ad7-8fe4-decd7009db3a"). InnerVolumeSpecName "kube-api-access-vjdfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:49:38.184888 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.184804    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjdfw\" (UniqueName: \"kubernetes.io/projected/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-kube-api-access-vjdfw\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:49:38.184888 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.184839    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:49:38.541918 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.541837    2578 generic.go:356] "Generic (PLEG): container finished" podID="dfd6b3fe-674b-4ad7-8fe4-decd7009db3a" containerID="33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806" exitCode=0
Sep 09 12:49:38.541918 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.541886    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"
Sep 09 12:49:38.542125 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.541928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7" event={"ID":"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a","Type":"ContainerDied","Data":"33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806"}
Sep 09 12:49:38.542125 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.541980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7" event={"ID":"dfd6b3fe-674b-4ad7-8fe4-decd7009db3a","Type":"ContainerDied","Data":"9e31c1b22258afe0448fc40c2278de7f3893b9c10737f857e06d91f65ec2b593"}
Sep 09 12:49:38.542125 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.542013    2578 scope.go:117] "RemoveContainer" containerID="33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806"
Sep 09 12:49:38.543448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.543429    2578 generic.go:356] "Generic (PLEG): container finished" podID="ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21" containerID="c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0" exitCode=0
Sep 09 12:49:38.543561 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.543451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh" event={"ID":"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21","Type":"ContainerDied","Data":"c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0"}
Sep 09 12:49:38.543561 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.543479    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"
Sep 09 12:49:38.543561 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.543500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh" event={"ID":"ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21","Type":"ContainerDied","Data":"1f562cbc996fd4e12cffcfd02e630956f212aa9a39ccdaad53b857a43e57bd9f"}
Sep 09 12:49:38.551368 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.551356    2578 scope.go:117] "RemoveContainer" containerID="33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806"
Sep 09 12:49:38.551653 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:49:38.551631    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806\": container with ID starting with 33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806 not found: ID does not exist" containerID="33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806"
Sep 09 12:49:38.551708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.551660    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806"} err="failed to get container status \"33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806\": rpc error: code = NotFound desc = could not find container \"33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806\": container with ID starting with 33c7a6fd896c45c95b6af1b851bc2b5f5021205fdb6e07108b623b05c883e806 not found: ID does not exist"
Sep 09 12:49:38.551708 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.551675    2578 scope.go:117] "RemoveContainer" containerID="c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0"
Sep 09 12:49:38.560128 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.560109    2578 scope.go:117] "RemoveContainer" containerID="c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0"
Sep 09 12:49:38.560426 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:49:38.560405    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0\": container with ID starting with c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0 not found: ID does not exist" containerID="c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0"
Sep 09 12:49:38.560503 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.560435    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0"} err="failed to get container status \"c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0\": rpc error: code = NotFound desc = could not find container \"c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0\": container with ID starting with c250cc4d0f448290479fc2701b0ca6622a95206a7852467c71d4761f677796e0 not found: ID does not exist"
Sep 09 12:49:38.561547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.561529    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"]
Sep 09 12:49:38.565083 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.565065    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2db0a6-76644d9d6-lsgp7"]
Sep 09 12:49:38.576929 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.576899    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"]
Sep 09 12:49:38.578609 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:38.578590    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2db0a6-58b6dc9ff-djcjh"]
Sep 09 12:49:40.186722 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:40.186690    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21" path="/var/lib/kubelet/pods/ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21/volumes"
Sep 09 12:49:40.187082 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:40.187035    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfd6b3fe-674b-4ad7-8fe4-decd7009db3a" path="/var/lib/kubelet/pods/dfd6b3fe-674b-4ad7-8fe4-decd7009db3a/volumes"
Sep 09 12:49:40.291951 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:40.291927    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:49:40.297353 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:40.297329    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:49:40.823634 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:40.823541    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:49:55.522095 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:55.522053    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"]
Sep 09 12:49:55.522751 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:55.522345    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx" podUID="4256461e-70dc-436f-84bf-707057b5e1a8" containerName="payload-processing" containerID="cri-o://b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3" gracePeriod=30
Sep 09 12:49:55.782366 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:55.782299    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:49:55.960740 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:55.960704    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7h4l\" (UniqueName: \"kubernetes.io/projected/4256461e-70dc-436f-84bf-707057b5e1a8-kube-api-access-l7h4l\") pod \"4256461e-70dc-436f-84bf-707057b5e1a8\" (UID: \"4256461e-70dc-436f-84bf-707057b5e1a8\") "
Sep 09 12:49:55.960911 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:55.960865    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4256461e-70dc-436f-84bf-707057b5e1a8-plugins-config-volume\") pod \"4256461e-70dc-436f-84bf-707057b5e1a8\" (UID: \"4256461e-70dc-436f-84bf-707057b5e1a8\") "
Sep 09 12:49:55.961245 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:55.961200    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4256461e-70dc-436f-84bf-707057b5e1a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4256461e-70dc-436f-84bf-707057b5e1a8" (UID: "4256461e-70dc-436f-84bf-707057b5e1a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:49:55.963032 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:55.963011    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4256461e-70dc-436f-84bf-707057b5e1a8-kube-api-access-l7h4l" (OuterVolumeSpecName: "kube-api-access-l7h4l") pod "4256461e-70dc-436f-84bf-707057b5e1a8" (UID: "4256461e-70dc-436f-84bf-707057b5e1a8"). InnerVolumeSpecName "kube-api-access-l7h4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:49:56.062035 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:56.061936    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4256461e-70dc-436f-84bf-707057b5e1a8-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:49:56.062035 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:56.061979    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7h4l\" (UniqueName: \"kubernetes.io/projected/4256461e-70dc-436f-84bf-707057b5e1a8-kube-api-access-l7h4l\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:49:56.637818 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:56.637779    2578 generic.go:356] "Generic (PLEG): container finished" podID="4256461e-70dc-436f-84bf-707057b5e1a8" containerID="b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3" exitCode=0
Sep 09 12:49:56.637818 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:56.637823    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"
Sep 09 12:49:56.638350 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:56.637864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx" event={"ID":"4256461e-70dc-436f-84bf-707057b5e1a8","Type":"ContainerDied","Data":"b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3"}
Sep 09 12:49:56.638350 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:56.637914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx" event={"ID":"4256461e-70dc-436f-84bf-707057b5e1a8","Type":"ContainerDied","Data":"b73aa6bac1a2606b6adb2b480e8bd86da415c96adab79cae2db6d51a1356877d"}
Sep 09 12:49:56.638350 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:56.637936    2578 scope.go:117] "RemoveContainer" containerID="b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3"
Sep 09 12:49:56.653872 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:56.653819    2578 scope.go:117] "RemoveContainer" containerID="b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3"
Sep 09 12:49:56.654205 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:49:56.654183    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3\": container with ID starting with b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3 not found: ID does not exist" containerID="b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3"
Sep 09 12:49:56.654316 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:56.654237    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3"} err="failed to get container status \"b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3\": rpc error: code = NotFound desc = could not find container \"b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3\": container with ID starting with b10af573c38060cd17eba904f5dfaf5385be010ee6bcf1235649c646d601fbc3 not found: ID does not exist"
Sep 09 12:49:56.655567 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:56.655527    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"]
Sep 09 12:49:56.657275 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:56.657257    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-283adb-84db579cff-2tzzx"]
Sep 09 12:49:58.186375 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:58.186344    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4256461e-70dc-436f-84bf-707057b5e1a8" path="/var/lib/kubelet/pods/4256461e-70dc-436f-84bf-707057b5e1a8/volumes"
Sep 09 12:49:58.430597 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:49:58.430560    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:50:13.729903 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:13.729869    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"]
Sep 09 12:50:13.730420 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:13.730119    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t" podUID="1de12030-3db6-45d3-8349-e9d856c1892a" containerName="maas-api" containerID="cri-o://5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4" gracePeriod=30
Sep 09 12:50:13.791600 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:13.791567    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"]
Sep 09 12:50:13.791841 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:13.791797    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6" podUID="0fddfae9-dfa3-487d-910a-5b3400648c38" containerName="payload-processing" containerID="cri-o://b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1" gracePeriod=30
Sep 09 12:50:14.034857 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.034831    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:50:14.067459 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.067434    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:50:14.135641 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.135608    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-maas-api-tls\") pod \"1de12030-3db6-45d3-8349-e9d856c1892a\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") "
Sep 09 12:50:14.135823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.135649    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4vfw\" (UniqueName: \"kubernetes.io/projected/1de12030-3db6-45d3-8349-e9d856c1892a-kube-api-access-m4vfw\") pod \"1de12030-3db6-45d3-8349-e9d856c1892a\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") "
Sep 09 12:50:14.135823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.135679    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fddfae9-dfa3-487d-910a-5b3400648c38-plugins-config-volume\") pod \"0fddfae9-dfa3-487d-910a-5b3400648c38\" (UID: \"0fddfae9-dfa3-487d-910a-5b3400648c38\") "
Sep 09 12:50:14.135823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.135729    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de12030-3db6-45d3-8349-e9d856c1892a-openshift-service-ca\") pod \"1de12030-3db6-45d3-8349-e9d856c1892a\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") "
Sep 09 12:50:14.135823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.135762    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cp4v7\" (UniqueName: \"kubernetes.io/projected/0fddfae9-dfa3-487d-910a-5b3400648c38-kube-api-access-cp4v7\") pod \"0fddfae9-dfa3-487d-910a-5b3400648c38\" (UID: \"0fddfae9-dfa3-487d-910a-5b3400648c38\") "
Sep 09 12:50:14.135823 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.135813    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-metrics-certs\") pod \"1de12030-3db6-45d3-8349-e9d856c1892a\" (UID: \"1de12030-3db6-45d3-8349-e9d856c1892a\") "
Sep 09 12:50:14.136140 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.136111    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de12030-3db6-45d3-8349-e9d856c1892a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1de12030-3db6-45d3-8349-e9d856c1892a" (UID: "1de12030-3db6-45d3-8349-e9d856c1892a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:50:14.136140 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.136119    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fddfae9-dfa3-487d-910a-5b3400648c38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0fddfae9-dfa3-487d-910a-5b3400648c38" (UID: "0fddfae9-dfa3-487d-910a-5b3400648c38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:50:14.137988 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.137962    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1de12030-3db6-45d3-8349-e9d856c1892a" (UID: "1de12030-3db6-45d3-8349-e9d856c1892a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:50:14.138115 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.138025    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fddfae9-dfa3-487d-910a-5b3400648c38-kube-api-access-cp4v7" (OuterVolumeSpecName: "kube-api-access-cp4v7") pod "0fddfae9-dfa3-487d-910a-5b3400648c38" (UID: "0fddfae9-dfa3-487d-910a-5b3400648c38"). InnerVolumeSpecName "kube-api-access-cp4v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:50:14.138115 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.138033    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de12030-3db6-45d3-8349-e9d856c1892a-kube-api-access-m4vfw" (OuterVolumeSpecName: "kube-api-access-m4vfw") pod "1de12030-3db6-45d3-8349-e9d856c1892a" (UID: "1de12030-3db6-45d3-8349-e9d856c1892a"). InnerVolumeSpecName "kube-api-access-m4vfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:50:14.138115 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.138054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1de12030-3db6-45d3-8349-e9d856c1892a" (UID: "1de12030-3db6-45d3-8349-e9d856c1892a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:50:14.236871 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.236842    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4vfw\" (UniqueName: \"kubernetes.io/projected/1de12030-3db6-45d3-8349-e9d856c1892a-kube-api-access-m4vfw\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:50:14.236871 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.236865    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fddfae9-dfa3-487d-910a-5b3400648c38-plugins-config-volume\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:50:14.237032 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.236875    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de12030-3db6-45d3-8349-e9d856c1892a-openshift-service-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:50:14.237032 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.236885    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp4v7\" (UniqueName: \"kubernetes.io/projected/0fddfae9-dfa3-487d-910a-5b3400648c38-kube-api-access-cp4v7\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:50:14.237032 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.236894    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-metrics-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:50:14.237032 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.236901    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de12030-3db6-45d3-8349-e9d856c1892a-maas-api-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:50:14.731307 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.731276    2578 generic.go:356] "Generic (PLEG): container finished" podID="1de12030-3db6-45d3-8349-e9d856c1892a" containerID="5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4" exitCode=0
Sep 09 12:50:14.731797 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.731325    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"
Sep 09 12:50:14.731797 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.731355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t" event={"ID":"1de12030-3db6-45d3-8349-e9d856c1892a","Type":"ContainerDied","Data":"5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4"}
Sep 09 12:50:14.731797 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.731415    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t" event={"ID":"1de12030-3db6-45d3-8349-e9d856c1892a","Type":"ContainerDied","Data":"5f1ac525ecb0bde111ef98173290e66e1d98e0f12544777e1daa37a259977a49"}
Sep 09 12:50:14.731797 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.731446    2578 scope.go:117] "RemoveContainer" containerID="5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4"
Sep 09 12:50:14.733093 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.733068    2578 generic.go:356] "Generic (PLEG): container finished" podID="0fddfae9-dfa3-487d-910a-5b3400648c38" containerID="b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1" exitCode=0
Sep 09 12:50:14.733168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.733112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6" event={"ID":"0fddfae9-dfa3-487d-910a-5b3400648c38","Type":"ContainerDied","Data":"b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1"}
Sep 09 12:50:14.733168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.733143    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"
Sep 09 12:50:14.733168 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.733154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6" event={"ID":"0fddfae9-dfa3-487d-910a-5b3400648c38","Type":"ContainerDied","Data":"c8efbca4d584dd0000f7b18713884224ef8ff1586e1010bc7dc30987ba0bf7d8"}
Sep 09 12:50:14.741048 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.741029    2578 scope.go:117] "RemoveContainer" containerID="5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4"
Sep 09 12:50:14.741362 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:50:14.741338    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4\": container with ID starting with 5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4 not found: ID does not exist" containerID="5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4"
Sep 09 12:50:14.741447 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.741369    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4"} err="failed to get container status \"5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4\": rpc error: code = NotFound desc = could not find container \"5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4\": container with ID starting with 5478623c1404e3bfba52f207d211336f6aa8f314256887a933f81511f4d591a4 not found: ID does not exist"
Sep 09 12:50:14.741447 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.741385    2578 scope.go:117] "RemoveContainer" containerID="b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1"
Sep 09 12:50:14.751578 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.751548    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"]
Sep 09 12:50:14.753326 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.753310    2578 scope.go:117] "RemoveContainer" containerID="b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1"
Sep 09 12:50:14.753677 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:50:14.753654    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1\": container with ID starting with b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1 not found: ID does not exist" containerID="b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1"
Sep 09 12:50:14.753749 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.753687    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1"} err="failed to get container status \"b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1\": rpc error: code = NotFound desc = could not find container \"b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1\": container with ID starting with b2e1d41d31464e36de4ccbd626fe438142cd6df99571a0bfb536ccff70af6ed1 not found: ID does not exist"
Sep 09 12:50:14.754451 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.754435    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe6606-6656c9477f-dvgp6"]
Sep 09 12:50:14.764009 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.763987    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"]
Sep 09 12:50:14.770301 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:14.770278    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe6606-78b9bcf6d8-tj56t"]
Sep 09 12:50:16.185749 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:16.185714    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fddfae9-dfa3-487d-910a-5b3400648c38" path="/var/lib/kubelet/pods/0fddfae9-dfa3-487d-910a-5b3400648c38/volumes"
Sep 09 12:50:16.186166 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:16.186088    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de12030-3db6-45d3-8349-e9d856c1892a" path="/var/lib/kubelet/pods/1de12030-3db6-45d3-8349-e9d856c1892a/volumes"
Sep 09 12:50:16.426658 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:16.426622    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:50:25.720610 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:50:25.720573    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:51:40.499835 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:51:40.499806    2578 scope.go:117] "RemoveContainer" containerID="6f28bc1ffee887dde6657be66913102a42a0c2eabc6fe24930d5b8b91e4515f6"
Sep 09 12:51:40.508433 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:51:40.508414    2578 scope.go:117] "RemoveContainer" containerID="f147a903108a08e0ccd2772037afe4d05023c2cdeb283e64ebc9c6cb91feeb17"
Sep 09 12:51:40.516324 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:51:40.516302    2578 scope.go:117] "RemoveContainer" containerID="cfcc0c5c479c82bf709afccb583a04462e74b3175cdf5163fc7917e4ef1c58bc"
Sep 09 12:51:47.322315 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:51:47.322281    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:52:13.324414 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:52:13.324382    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:52:21.430054 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:52:21.430015    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:52:30.620916 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:52:30.620869    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:52:46.924633 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:52:46.924556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:52:53.223820 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:52:53.223778    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:54:18.928797 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:18.928763    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:54:27.928811 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:27.928772    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:54:37.328912 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:37.328878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:54:40.324435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:40.324400    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:54:40.331716 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:40.331695    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:54:43.420731 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.420699    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"]
Sep 09 12:54:43.421130 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421115    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd6b3fe-674b-4ad7-8fe4-decd7009db3a" containerName="payload-pre-processing"
Sep 09 12:54:43.421175 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421133    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd6b3fe-674b-4ad7-8fe4-decd7009db3a" containerName="payload-pre-processing"
Sep 09 12:54:43.421175 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421147    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21" containerName="maas-api"
Sep 09 12:54:43.421175 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421153    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21" containerName="maas-api"
Sep 09 12:54:43.421175 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421167    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de12030-3db6-45d3-8349-e9d856c1892a" containerName="maas-api"
Sep 09 12:54:43.421175 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421172    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de12030-3db6-45d3-8349-e9d856c1892a" containerName="maas-api"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421182    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67bfbe6f-43c9-48fd-b871-8d9670660502" containerName="maas-api"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421187    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67bfbe6f-43c9-48fd-b871-8d9670660502" containerName="maas-api"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421199    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4256461e-70dc-436f-84bf-707057b5e1a8" containerName="payload-processing"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421204    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4256461e-70dc-436f-84bf-707057b5e1a8" containerName="payload-processing"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421244    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fddfae9-dfa3-487d-910a-5b3400648c38" containerName="payload-processing"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421250    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fddfae9-dfa3-487d-910a-5b3400648c38" containerName="payload-processing"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421256    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbf20340-1e39-430b-adea-a224f7557aaf" containerName="payload-pre-processing"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421262    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbf20340-1e39-430b-adea-a224f7557aaf" containerName="payload-pre-processing"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421325    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de12030-3db6-45d3-8349-e9d856c1892a" containerName="maas-api"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421334    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5dd6e7-ad9c-4f0d-b5f4-59bd0b6a1c21" containerName="maas-api"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421340    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbf20340-1e39-430b-adea-a224f7557aaf" containerName="payload-pre-processing"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421346    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4256461e-70dc-436f-84bf-707057b5e1a8" containerName="payload-processing"
Sep 09 12:54:43.421352 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421353    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfd6b3fe-674b-4ad7-8fe4-decd7009db3a" containerName="payload-pre-processing"
Sep 09 12:54:43.421739 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421360    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fddfae9-dfa3-487d-910a-5b3400648c38" containerName="payload-processing"
Sep 09 12:54:43.421739 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.421365    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67bfbe6f-43c9-48fd-b871-8d9670660502" containerName="maas-api"
Sep 09 12:54:43.424105 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.424089    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.426008 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.425986    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8e37d975-kserve-self-signed-certs\""
Sep 09 12:54:43.432903 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.432879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"]
Sep 09 12:54:43.561592 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.561550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f29bc4d-be93-4907-a9b5-14f52b098257-tls-certs\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.561592 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.561590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-model-cache\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.561863 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.561631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-tmp-dir\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.561863 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.561753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-dshm\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.561863 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.561797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8np2d\" (UniqueName: \"kubernetes.io/projected/3f29bc4d-be93-4907-a9b5-14f52b098257-kube-api-access-8np2d\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.561863 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.561862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-home\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.662601 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.662562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-dshm\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.662795 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.662613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8np2d\" (UniqueName: \"kubernetes.io/projected/3f29bc4d-be93-4907-a9b5-14f52b098257-kube-api-access-8np2d\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.662795 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.662648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-home\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.662795 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.662690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f29bc4d-be93-4907-a9b5-14f52b098257-tls-certs\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.662795 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.662717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-model-cache\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.662795 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.662773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-tmp-dir\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.663074 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.663016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-home\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.663136 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.663075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-model-cache\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.663188 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.663139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-tmp-dir\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.664961 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.664940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-dshm\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.665258 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.665205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f29bc4d-be93-4907-a9b5-14f52b098257-tls-certs\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.670333 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.670313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8np2d\" (UniqueName: \"kubernetes.io/projected/3f29bc4d-be93-4907-a9b5-14f52b098257-kube-api-access-8np2d\") pod \"e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") " pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:43.734482 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:43.734397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:44.010067 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:44.010041    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"]
Sep 09 12:54:44.012188 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:54:44.012150    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f29bc4d_be93_4907_a9b5_14f52b098257.slice/crio-ee3989eeba65ac06750f08363ca4044db635673d75996905e18d03d0d2b566bf WatchSource:0}: Error finding container ee3989eeba65ac06750f08363ca4044db635673d75996905e18d03d0d2b566bf: Status 404 returned error can't find the container with id ee3989eeba65ac06750f08363ca4044db635673d75996905e18d03d0d2b566bf
Sep 09 12:54:44.044784 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:44.044756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52" event={"ID":"3f29bc4d-be93-4907-a9b5-14f52b098257","Type":"ContainerStarted","Data":"ee3989eeba65ac06750f08363ca4044db635673d75996905e18d03d0d2b566bf"}
Sep 09 12:54:45.050857 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:45.050820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52" event={"ID":"3f29bc4d-be93-4907-a9b5-14f52b098257","Type":"ContainerStarted","Data":"14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7"}
Sep 09 12:54:45.067702 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:45.067653    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52" podStartSLOduration=2.067618284 podStartE2EDuration="2.067618284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:54:45.065877086 +0000 UTC m=+2105.409475033" watchObservedRunningTime="2026-09-09 12:54:45.067618284 +0000 UTC m=+2105.411216231"
Sep 09 12:54:47.232916 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:47.232882    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:54:53.735056 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:53.735005    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:53.735056 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:53.735067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:53.747835 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:53.747813    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:54:54.106409 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:54:54.106320    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:55:22.622645 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:22.622613    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"]
Sep 09 12:55:25.426410 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:25.426372    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:55:35.031348 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:35.031312    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:55:37.643561 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:37.643496    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52" podUID="3f29bc4d-be93-4907-a9b5-14f52b098257" containerName="main" containerID="cri-o://14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7" gracePeriod=45
Sep 09 12:55:37.885955 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:37.885930    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:55:38.001707 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.001670    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-tmp-dir\") pod \"3f29bc4d-be93-4907-a9b5-14f52b098257\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") "
Sep 09 12:55:38.001926 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.001729    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f29bc4d-be93-4907-a9b5-14f52b098257-tls-certs\") pod \"3f29bc4d-be93-4907-a9b5-14f52b098257\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") "
Sep 09 12:55:38.001926 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.001760    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8np2d\" (UniqueName: \"kubernetes.io/projected/3f29bc4d-be93-4907-a9b5-14f52b098257-kube-api-access-8np2d\") pod \"3f29bc4d-be93-4907-a9b5-14f52b098257\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") "
Sep 09 12:55:38.001926 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.001799    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-home\") pod \"3f29bc4d-be93-4907-a9b5-14f52b098257\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") "
Sep 09 12:55:38.001926 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.001813    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-dshm\") pod \"3f29bc4d-be93-4907-a9b5-14f52b098257\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") "
Sep 09 12:55:38.001926 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.001847    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-model-cache\") pod \"3f29bc4d-be93-4907-a9b5-14f52b098257\" (UID: \"3f29bc4d-be93-4907-a9b5-14f52b098257\") "
Sep 09 12:55:38.002209 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.002004    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f29bc4d-be93-4907-a9b5-14f52b098257" (UID: "3f29bc4d-be93-4907-a9b5-14f52b098257"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:55:38.002209 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.002111    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-home" (OuterVolumeSpecName: "home") pod "3f29bc4d-be93-4907-a9b5-14f52b098257" (UID: "3f29bc4d-be93-4907-a9b5-14f52b098257"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:55:38.002209 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.002130    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f29bc4d-be93-4907-a9b5-14f52b098257" (UID: "3f29bc4d-be93-4907-a9b5-14f52b098257"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:55:38.002209 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.002190    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-tmp-dir\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:55:38.004015 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.003983    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-dshm" (OuterVolumeSpecName: "dshm") pod "3f29bc4d-be93-4907-a9b5-14f52b098257" (UID: "3f29bc4d-be93-4907-a9b5-14f52b098257"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:55:38.004112 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.004049    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f29bc4d-be93-4907-a9b5-14f52b098257-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f29bc4d-be93-4907-a9b5-14f52b098257" (UID: "3f29bc4d-be93-4907-a9b5-14f52b098257"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:55:38.004243 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.004203    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f29bc4d-be93-4907-a9b5-14f52b098257-kube-api-access-8np2d" (OuterVolumeSpecName: "kube-api-access-8np2d") pod "3f29bc4d-be93-4907-a9b5-14f52b098257" (UID: "3f29bc4d-be93-4907-a9b5-14f52b098257"). InnerVolumeSpecName "kube-api-access-8np2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:55:38.102870 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.102836    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8np2d\" (UniqueName: \"kubernetes.io/projected/3f29bc4d-be93-4907-a9b5-14f52b098257-kube-api-access-8np2d\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:55:38.102870 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.102864    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-home\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:55:38.102870 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.102873    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-dshm\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:55:38.102870 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.102881    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f29bc4d-be93-4907-a9b5-14f52b098257-model-cache\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:55:38.103140 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.102889    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f29bc4d-be93-4907-a9b5-14f52b098257-tls-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:55:38.316800 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.316707    2578 generic.go:356] "Generic (PLEG): container finished" podID="3f29bc4d-be93-4907-a9b5-14f52b098257" containerID="14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7" exitCode=0
Sep 09 12:55:38.316800 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.316768    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"
Sep 09 12:55:38.316800 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.316788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52" event={"ID":"3f29bc4d-be93-4907-a9b5-14f52b098257","Type":"ContainerDied","Data":"14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7"}
Sep 09 12:55:38.317153 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.316835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52" event={"ID":"3f29bc4d-be93-4907-a9b5-14f52b098257","Type":"ContainerDied","Data":"ee3989eeba65ac06750f08363ca4044db635673d75996905e18d03d0d2b566bf"}
Sep 09 12:55:38.317153 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.316852    2578 scope.go:117] "RemoveContainer" containerID="14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7"
Sep 09 12:55:38.325615 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.325596    2578 scope.go:117] "RemoveContainer" containerID="14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7"
Sep 09 12:55:38.325906 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:55:38.325886    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7\": container with ID starting with 14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7 not found: ID does not exist" containerID="14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7"
Sep 09 12:55:38.325955 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.325918    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7"} err="failed to get container status \"14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7\": rpc error: code = NotFound desc = could not find container \"14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7\": container with ID starting with 14cb23b358e97d182b4c2c44ae79ff6481702bf42aed1d734f241cc2c5fc36b7 not found: ID does not exist"
Sep 09 12:55:38.332380 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.332359    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"]
Sep 09 12:55:38.338114 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:38.338094    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8e37d975-kserve-5ccbf8dcbc-vkd52"]
Sep 09 12:55:40.186649 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:40.186618    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f29bc4d-be93-4907-a9b5-14f52b098257" path="/var/lib/kubelet/pods/3f29bc4d-be93-4907-a9b5-14f52b098257/volumes"
Sep 09 12:55:41.725256 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:41.725204    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:55:42.138249 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.138149    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"]
Sep 09 12:55:42.138586 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.138573    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f29bc4d-be93-4907-a9b5-14f52b098257" containerName="main"
Sep 09 12:55:42.138632 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.138588    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f29bc4d-be93-4907-a9b5-14f52b098257" containerName="main"
Sep 09 12:55:42.138669 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.138658    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f29bc4d-be93-4907-a9b5-14f52b098257" containerName="main"
Sep 09 12:55:42.141500 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.141485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.143413 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.143388    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 12:55:42.144011 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.143992    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5rdxz\""
Sep 09 12:55:42.144106 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.144028    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 12:55:42.144106 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.144034    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 12:55:42.144180 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.144060    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 12:55:42.154410 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.154385    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"]
Sep 09 12:55:42.243386 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.243349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-maas-api-tls\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.243579 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.243449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-metrics-certs\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.243579 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.243477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h745d\" (UniqueName: \"kubernetes.io/projected/8ffe2bdb-6a0c-46b7-95ed-90796656753f-kube-api-access-h745d\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.243579 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.243562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe2bdb-6a0c-46b7-95ed-90796656753f-openshift-service-ca\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.344139 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.344102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe2bdb-6a0c-46b7-95ed-90796656753f-openshift-service-ca\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.344322 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.344153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-maas-api-tls\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.344322 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.344273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-metrics-certs\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.344322 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.344314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h745d\" (UniqueName: \"kubernetes.io/projected/8ffe2bdb-6a0c-46b7-95ed-90796656753f-kube-api-access-h745d\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.344762 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.344732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe2bdb-6a0c-46b7-95ed-90796656753f-openshift-service-ca\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.346760 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.346739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-maas-api-tls\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.346856 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.346782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-metrics-certs\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.351820 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.351800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h745d\" (UniqueName: \"kubernetes.io/projected/8ffe2bdb-6a0c-46b7-95ed-90796656753f-kube-api-access-h745d\") pod \"maas-api-59675b95bf-9lrnk\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") " pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.451435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.451403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:42.722759 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.722697    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"]
Sep 09 12:55:42.725018 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:55:42.724987    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffe2bdb_6a0c_46b7_95ed_90796656753f.slice/crio-4c2f4240a6f593f45d6bb59f7f716d7e279b981ee7279aca8f16dd5ce2b562ff WatchSource:0}: Error finding container 4c2f4240a6f593f45d6bb59f7f716d7e279b981ee7279aca8f16dd5ce2b562ff: Status 404 returned error can't find the container with id 4c2f4240a6f593f45d6bb59f7f716d7e279b981ee7279aca8f16dd5ce2b562ff
Sep 09 12:55:42.726333 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:42.726312    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:55:43.346699 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:43.346659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk" event={"ID":"8ffe2bdb-6a0c-46b7-95ed-90796656753f","Type":"ContainerStarted","Data":"823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca"}
Sep 09 12:55:43.346819 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:43.346772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:43.346819 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:43.346802    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk" event={"ID":"8ffe2bdb-6a0c-46b7-95ed-90796656753f","Type":"ContainerStarted","Data":"4c2f4240a6f593f45d6bb59f7f716d7e279b981ee7279aca8f16dd5ce2b562ff"}
Sep 09 12:55:43.364278 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:43.364192    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk" podStartSLOduration=0.818111455 podStartE2EDuration="1.364166694s" totalImagesPullingTime="546.055239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:55:43.362269769 +0000 UTC m=+2163.705867708" watchObservedRunningTime="2026-09-09 12:55:43.364166694 +0000 UTC m=+2163.707764641"
Sep 09 12:55:49.358089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.358060    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:49.626617 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.626524    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"]
Sep 09 12:55:49.630234 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.630195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.647287 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.647257    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"]
Sep 09 12:55:49.718745 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.718707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be387196-5685-4dcf-a400-1f3127e7f402-metrics-certs\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.718914 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.718824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tptmv\" (UniqueName: \"kubernetes.io/projected/be387196-5685-4dcf-a400-1f3127e7f402-kube-api-access-tptmv\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.718914 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.718846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be387196-5685-4dcf-a400-1f3127e7f402-openshift-service-ca\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.719010 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.718912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be387196-5685-4dcf-a400-1f3127e7f402-maas-api-tls\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.727626 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.727602    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:55:49.819508 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.819477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tptmv\" (UniqueName: \"kubernetes.io/projected/be387196-5685-4dcf-a400-1f3127e7f402-kube-api-access-tptmv\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.819508 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.819512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be387196-5685-4dcf-a400-1f3127e7f402-openshift-service-ca\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.819758 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.819578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be387196-5685-4dcf-a400-1f3127e7f402-maas-api-tls\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.819758 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.819612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be387196-5685-4dcf-a400-1f3127e7f402-metrics-certs\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.820317 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.820287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be387196-5685-4dcf-a400-1f3127e7f402-openshift-service-ca\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.822239 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.822190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be387196-5685-4dcf-a400-1f3127e7f402-metrics-certs\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.822357 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.822250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be387196-5685-4dcf-a400-1f3127e7f402-maas-api-tls\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.827771 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.827748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tptmv\" (UniqueName: \"kubernetes.io/projected/be387196-5685-4dcf-a400-1f3127e7f402-kube-api-access-tptmv\") pod \"maas-api-8df5f485c-gl9zg\" (UID: \"be387196-5685-4dcf-a400-1f3127e7f402\") " pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:49.940980 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:49.940935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:50.212735 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:50.212710    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"]
Sep 09 12:55:50.215089 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:55:50.215058    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe387196_5685_4dcf_a400_1f3127e7f402.slice/crio-5fddd1063f69b7a62d5ce01a5bf2ff57ecbf0d540e50bdd2a309e2426f741cf3 WatchSource:0}: Error finding container 5fddd1063f69b7a62d5ce01a5bf2ff57ecbf0d540e50bdd2a309e2426f741cf3: Status 404 returned error can't find the container with id 5fddd1063f69b7a62d5ce01a5bf2ff57ecbf0d540e50bdd2a309e2426f741cf3
Sep 09 12:55:50.384431 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:50.384385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg" event={"ID":"be387196-5685-4dcf-a400-1f3127e7f402","Type":"ContainerStarted","Data":"5fddd1063f69b7a62d5ce01a5bf2ff57ecbf0d540e50bdd2a309e2426f741cf3"}
Sep 09 12:55:51.390584 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:51.390553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg" event={"ID":"be387196-5685-4dcf-a400-1f3127e7f402","Type":"ContainerStarted","Data":"b991af67ba211e20ec6c77a34bb79d71d121f6b5adf104ce30aad81ffa7a059e"}
Sep 09 12:55:51.390964 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:51.390679    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:51.412151 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:51.412107    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg" podStartSLOduration=2.023201866 podStartE2EDuration="2.412090521s" totalImagesPullingTime="388.888655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:55:51.409778094 +0000 UTC m=+2171.753376042" watchObservedRunningTime="2026-09-09 12:55:51.412090521 +0000 UTC m=+2171.755688469"
Sep 09 12:55:57.031577 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:57.031534    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:55:57.400939 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:57.400915    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8df5f485c-gl9zg"
Sep 09 12:55:57.451787 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:57.451744    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"]
Sep 09 12:55:57.452060 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:57.452032    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk" podUID="8ffe2bdb-6a0c-46b7-95ed-90796656753f" containerName="maas-api" containerID="cri-o://823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca" gracePeriod=30
Sep 09 12:55:58.704415 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.704386    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:58.812575 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.812481    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h745d\" (UniqueName: \"kubernetes.io/projected/8ffe2bdb-6a0c-46b7-95ed-90796656753f-kube-api-access-h745d\") pod \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") "
Sep 09 12:55:58.812750 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.812588    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe2bdb-6a0c-46b7-95ed-90796656753f-openshift-service-ca\") pod \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") "
Sep 09 12:55:58.812750 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.812654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-maas-api-tls\") pod \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") "
Sep 09 12:55:58.812750 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.812680    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-metrics-certs\") pod \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\" (UID: \"8ffe2bdb-6a0c-46b7-95ed-90796656753f\") "
Sep 09 12:55:58.813017 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.812987    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ffe2bdb-6a0c-46b7-95ed-90796656753f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ffe2bdb-6a0c-46b7-95ed-90796656753f" (UID: "8ffe2bdb-6a0c-46b7-95ed-90796656753f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:55:58.814839 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.814811    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ffe2bdb-6a0c-46b7-95ed-90796656753f-kube-api-access-h745d" (OuterVolumeSpecName: "kube-api-access-h745d") pod "8ffe2bdb-6a0c-46b7-95ed-90796656753f" (UID: "8ffe2bdb-6a0c-46b7-95ed-90796656753f"). InnerVolumeSpecName "kube-api-access-h745d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:55:58.814948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.814873    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ffe2bdb-6a0c-46b7-95ed-90796656753f" (UID: "8ffe2bdb-6a0c-46b7-95ed-90796656753f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:55:58.814948 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.814900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8ffe2bdb-6a0c-46b7-95ed-90796656753f" (UID: "8ffe2bdb-6a0c-46b7-95ed-90796656753f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:55:58.913518 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.913485    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h745d\" (UniqueName: \"kubernetes.io/projected/8ffe2bdb-6a0c-46b7-95ed-90796656753f-kube-api-access-h745d\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:55:58.913518 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.913510    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe2bdb-6a0c-46b7-95ed-90796656753f-openshift-service-ca\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:55:58.913518 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.913520    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-maas-api-tls\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:55:58.913518 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:58.913528    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ffe2bdb-6a0c-46b7-95ed-90796656753f-metrics-certs\") on node \"ip-10-0-133-152.ec2.internal\" DevicePath \"\""
Sep 09 12:55:59.438375 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:59.438333    2578 generic.go:356] "Generic (PLEG): container finished" podID="8ffe2bdb-6a0c-46b7-95ed-90796656753f" containerID="823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca" exitCode=0
Sep 09 12:55:59.438581 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:59.438533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk" event={"ID":"8ffe2bdb-6a0c-46b7-95ed-90796656753f","Type":"ContainerDied","Data":"823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca"}
Sep 09 12:55:59.438643 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:59.438580    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"
Sep 09 12:55:59.438643 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:59.438620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk" event={"ID":"8ffe2bdb-6a0c-46b7-95ed-90796656753f","Type":"ContainerDied","Data":"4c2f4240a6f593f45d6bb59f7f716d7e279b981ee7279aca8f16dd5ce2b562ff"}
Sep 09 12:55:59.438729 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:59.438648    2578 scope.go:117] "RemoveContainer" containerID="823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca"
Sep 09 12:55:59.448342 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:59.448324    2578 scope.go:117] "RemoveContainer" containerID="823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca"
Sep 09 12:55:59.448593 ip-10-0-133-152 kubenswrapper[2578]: E0909 12:55:59.448577    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca\": container with ID starting with 823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca not found: ID does not exist" containerID="823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca"
Sep 09 12:55:59.448639 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:59.448600    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca"} err="failed to get container status \"823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca\": rpc error: code = NotFound desc = could not find container \"823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca\": container with ID starting with 823acb64a1fb0ad36f2889b6c7d894e11f4d53f1abeeed7b85667d9ff2ba5eca not found: ID does not exist"
Sep 09 12:55:59.460102 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:59.460069    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"]
Sep 09 12:55:59.463788 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:55:59.463759    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59675b95bf-9lrnk"]
Sep 09 12:56:00.186559 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:00.186527    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ffe2bdb-6a0c-46b7-95ed-90796656753f" path="/var/lib/kubelet/pods/8ffe2bdb-6a0c-46b7-95ed-90796656753f/volumes"
Sep 09 12:56:05.322917 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:05.322882    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:56:10.528612 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:10.528578    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jrc6g"]
Sep 09 12:56:19.306837 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:19.306806    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5d49f968-hfkz2_d2536917-17ea-419b-84ee-a01daefd69e7/copy-manifests/0.log"
Sep 09 12:56:19.313678 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:19.313657    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5d49f968-hfkz2_d2536917-17ea-419b-84ee-a01daefd69e7/manager/0.log"
Sep 09 12:56:20.058072 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:20.058040    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8df5f485c-gl9zg_be387196-5685-4dcf-a400-1f3127e7f402/maas-api/0.log"
Sep 09 12:56:21.453564 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:21.453534    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k6t92_9ed9b5d1-05db-4345-8dd6-945c00d4b1a0/kuadrant-console-plugin/0.log"
Sep 09 12:56:21.708603 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:21.708527    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jrc6g_4451aeda-66e4-4ad3-a46a-5084e4e7a738/limitador/0.log"
Sep 09 12:56:22.425653 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:22.425623    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5948bbb55b-6bs6z_c0cfa3ac-1334-4119-9c97-da15303f86ba/kube-auth-proxy/0.log"
Sep 09 12:56:23.021196 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:23.021162    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8c9778b-t8sr5_11252896-9746-4dbf-9ca6-01c7d854fab9/router/0.log"
Sep 09 12:56:23.732807 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:23.732778    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-8nxhq_99e5f97e-ed4b-44a2-b4f1-0ac45e61f8e4/main/0.log"
Sep 09 12:56:23.983435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:23.983355    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-88k2b_13c383e6-88a2-44ab-a757-e484818d9b04/main/0.log"
Sep 09 12:56:24.097980 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:24.097947    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6c8q4_8d17903d-f84a-40ec-8289-a9a431ddec10/main/0.log"
Sep 09 12:56:32.946769 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:32.946740    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qdgrt_d09d81f3-b038-48e2-88c2-6ab36ddefce2/global-pull-secret-syncer/0.log"
Sep 09 12:56:33.034620 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:33.034590    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jf5dm_4f087b26-7171-44d4-a216-7c223ebc6e4b/konnectivity-agent/0.log"
Sep 09 12:56:33.089502 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:33.089475    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-152.ec2.internal_a3308eb29717fa7fe5c5ea5d97019ea1/haproxy/0.log"
Sep 09 12:56:37.749574 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:37.749545    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k6t92_9ed9b5d1-05db-4345-8dd6-945c00d4b1a0/kuadrant-console-plugin/0.log"
Sep 09 12:56:37.818394 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:37.818366    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jrc6g_4451aeda-66e4-4ad3-a46a-5084e4e7a738/limitador/0.log"
Sep 09 12:56:39.375693 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:39.375617    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a290f195-8dc2-418c-8b43-67515c1e4eed/alertmanager/0.log"
Sep 09 12:56:39.414324 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:39.414295    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a290f195-8dc2-418c-8b43-67515c1e4eed/config-reloader/0.log"
Sep 09 12:56:39.458195 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:39.458167    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a290f195-8dc2-418c-8b43-67515c1e4eed/kube-rbac-proxy-web/0.log"
Sep 09 12:56:39.497484 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:39.497459    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a290f195-8dc2-418c-8b43-67515c1e4eed/kube-rbac-proxy/0.log"
Sep 09 12:56:39.536246 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:39.536202    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a290f195-8dc2-418c-8b43-67515c1e4eed/kube-rbac-proxy-metric/0.log"
Sep 09 12:56:39.578083 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:39.578060    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a290f195-8dc2-418c-8b43-67515c1e4eed/prom-label-proxy/0.log"
Sep 09 12:56:39.622105 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:39.622081    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a290f195-8dc2-418c-8b43-67515c1e4eed/init-config-reloader/0.log"
Sep 09 12:56:39.679080 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:39.679051    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-rc7bk_0f9a503b-3e0d-4018-a745-aab35e92648f/cluster-monitoring-operator/0.log"
Sep 09 12:56:39.803435 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:39.803408    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6c9c7b7756-5smfc_438d1315-9ce0-40cc-8238-fda2188b0eba/metrics-server/0.log"
Sep 09 12:56:40.002921 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.002854    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm2bh_9c1b593e-ca9f-415e-884d-80d63e65d5e1/node-exporter/0.log"
Sep 09 12:56:40.059478 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.059412    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm2bh_9c1b593e-ca9f-415e-884d-80d63e65d5e1/kube-rbac-proxy/0.log"
Sep 09 12:56:40.087460 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.087436    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm2bh_9c1b593e-ca9f-415e-884d-80d63e65d5e1/init-textfile/0.log"
Sep 09 12:56:40.280919 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.280840    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_023c9c13-178a-43e2-bdc8-11ff23633d15/prometheus/0.log"
Sep 09 12:56:40.305111 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.305089    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_023c9c13-178a-43e2-bdc8-11ff23633d15/config-reloader/0.log"
Sep 09 12:56:40.332358 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.332337    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_023c9c13-178a-43e2-bdc8-11ff23633d15/thanos-sidecar/0.log"
Sep 09 12:56:40.355726 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.355705    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_023c9c13-178a-43e2-bdc8-11ff23633d15/kube-rbac-proxy-web/0.log"
Sep 09 12:56:40.378399 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.378379    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_023c9c13-178a-43e2-bdc8-11ff23633d15/kube-rbac-proxy/0.log"
Sep 09 12:56:40.402639 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.402610    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_023c9c13-178a-43e2-bdc8-11ff23633d15/kube-rbac-proxy-thanos/0.log"
Sep 09 12:56:40.425065 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.425044    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_023c9c13-178a-43e2-bdc8-11ff23633d15/init-config-reloader/0.log"
Sep 09 12:56:40.511045 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.511018    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-cn2x4_b2e52117-b695-4db3-95e1-be46d4f2a998/prometheus-operator-admission-webhook/0.log"
Sep 09 12:56:40.645354 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.645333    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-689c6d9ddd-rn82x_ff25c9f1-6d1d-438b-affd-f726ea7e7281/thanos-query/0.log"
Sep 09 12:56:40.672417 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.672391    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-689c6d9ddd-rn82x_ff25c9f1-6d1d-438b-affd-f726ea7e7281/kube-rbac-proxy-web/0.log"
Sep 09 12:56:40.694318 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.694299    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-689c6d9ddd-rn82x_ff25c9f1-6d1d-438b-affd-f726ea7e7281/kube-rbac-proxy/0.log"
Sep 09 12:56:40.719392 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.719373    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-689c6d9ddd-rn82x_ff25c9f1-6d1d-438b-affd-f726ea7e7281/prom-label-proxy/0.log"
Sep 09 12:56:40.757037 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.757014    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-689c6d9ddd-rn82x_ff25c9f1-6d1d-438b-affd-f726ea7e7281/kube-rbac-proxy-rules/0.log"
Sep 09 12:56:40.779798 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:40.779775    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-689c6d9ddd-rn82x_ff25c9f1-6d1d-438b-affd-f726ea7e7281/kube-rbac-proxy-metrics/0.log"
Sep 09 12:56:42.239876 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.239845    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-4ch9d_6cb90a7b-a57f-4dc3-9231-c74b11e6173e/networking-console-plugin/0.log"
Sep 09 12:56:42.862036 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.862005    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"]
Sep 09 12:56:42.862407 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.862395    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffe2bdb-6a0c-46b7-95ed-90796656753f" containerName="maas-api"
Sep 09 12:56:42.862458 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.862408    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffe2bdb-6a0c-46b7-95ed-90796656753f" containerName="maas-api"
Sep 09 12:56:42.862494 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.862478    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ffe2bdb-6a0c-46b7-95ed-90796656753f" containerName="maas-api"
Sep 09 12:56:42.867063 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.867045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:42.869545 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.869521    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-prpjx\"/\"openshift-service-ca.crt\""
Sep 09 12:56:42.869660 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.869575    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-prpjx\"/\"default-dockercfg-lz68r\""
Sep 09 12:56:42.869660 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.869525    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-prpjx\"/\"kube-root-ca.crt\""
Sep 09 12:56:42.873867 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.873839    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"]
Sep 09 12:56:42.886200 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.886171    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/2.log"
Sep 09 12:56:42.891424 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:42.891404    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hzzgw_0548208b-e765-4614-b647-363d001d99c4/console-operator/3.log"
Sep 09 12:56:43.031230 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.031186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-podres\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.031230 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.031228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-lib-modules\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.031439 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.031256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-sys\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.031439 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.031319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89brc\" (UniqueName: \"kubernetes.io/projected/f7b782c4-340e-4444-a8e6-a24d023c052c-kube-api-access-89brc\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.031439 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.031362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-proc\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.132736 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.132648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-podres\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.132736 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.132691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-lib-modules\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.132736 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.132723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-sys\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.132962 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.132753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89brc\" (UniqueName: \"kubernetes.io/projected/f7b782c4-340e-4444-a8e6-a24d023c052c-kube-api-access-89brc\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.132962 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.132786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-proc\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.132962 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.132835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-sys\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.132962 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.132835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-podres\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.132962 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.132875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-proc\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.132962 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.132876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7b782c4-340e-4444-a8e6-a24d023c052c-lib-modules\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.140459 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.140431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89brc\" (UniqueName: \"kubernetes.io/projected/f7b782c4-340e-4444-a8e6-a24d023c052c-kube-api-access-89brc\") pod \"perf-node-gather-daemonset-9ncvj\" (UID: \"f7b782c4-340e-4444-a8e6-a24d023c052c\") " pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.177877 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.177848    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.447954 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.447900    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"]
Sep 09 12:56:43.450185 ip-10-0-133-152 kubenswrapper[2578]: W0909 12:56:43.450155    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf7b782c4_340e_4444_a8e6_a24d023c052c.slice/crio-4b30852027d257e5b3f0b6f4185b95086a29d2723bf18db9b0804967158e3026 WatchSource:0}: Error finding container 4b30852027d257e5b3f0b6f4185b95086a29d2723bf18db9b0804967158e3026: Status 404 returned error can't find the container with id 4b30852027d257e5b3f0b6f4185b95086a29d2723bf18db9b0804967158e3026
Sep 09 12:56:43.655657 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.655622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj" event={"ID":"f7b782c4-340e-4444-a8e6-a24d023c052c","Type":"ContainerStarted","Data":"8add75825e464c7c31f6939957450af16442fcc40c228d74e65e17ad4f257135"}
Sep 09 12:56:43.655824 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.655686    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:43.655824 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.655703    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj" event={"ID":"f7b782c4-340e-4444-a8e6-a24d023c052c","Type":"ContainerStarted","Data":"4b30852027d257e5b3f0b6f4185b95086a29d2723bf18db9b0804967158e3026"}
Sep 09 12:56:43.671919 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:43.671871    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj" podStartSLOduration=1.671857511 podStartE2EDuration="1.671857511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:56:43.669643947 +0000 UTC m=+2224.013241907" watchObservedRunningTime="2026-09-09 12:56:43.671857511 +0000 UTC m=+2224.015455459"
Sep 09 12:56:44.188527 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:44.188495    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-xxtvw_b538ac89-4973-401d-aedf-67b03ff11d0a/volume-data-source-validator/0.log"
Sep 09 12:56:45.352089 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:45.352063    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f95hr_fd502c7b-b814-4afd-8516-6d6e7fa25979/dns/0.log"
Sep 09 12:56:45.374448 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:45.374414    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f95hr_fd502c7b-b814-4afd-8516-6d6e7fa25979/kube-rbac-proxy/0.log"
Sep 09 12:56:45.501104 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:45.501076    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zhgxx_b2b7fd24-9e5e-488e-aa9a-62a5531e9ab3/dns-node-resolver/0.log"
Sep 09 12:56:46.107793 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:46.107763    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t7kpc_6bb40332-7f47-480d-8fef-7be82e907b6a/node-ca/0.log"
Sep 09 12:56:47.574484 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:47.574453    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5948bbb55b-6bs6z_c0cfa3ac-1334-4119-9c97-da15303f86ba/kube-auth-proxy/0.log"
Sep 09 12:56:47.750358 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:47.750333    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8c9778b-t8sr5_11252896-9746-4dbf-9ca6-01c7d854fab9/router/0.log"
Sep 09 12:56:48.606300 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:48.606270    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-88dzs_d7e96ad5-f31e-45dd-b44e-e49453638364/serve-healthcheck-canary/0.log"
Sep 09 12:56:49.152406 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:49.152378    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-tpw62_ea6a22dd-7c89-4083-831d-3598aeb4bf58/insights-operator/0.log"
Sep 09 12:56:49.152790 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:49.152775    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-tpw62_ea6a22dd-7c89-4083-831d-3598aeb4bf58/insights-operator/1.log"
Sep 09 12:56:49.318439 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:49.318411    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vl2k6_3c86c595-fb8e-45e3-b6c0-cb9871af938d/kube-rbac-proxy/0.log"
Sep 09 12:56:49.341154 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:49.341122    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vl2k6_3c86c595-fb8e-45e3-b6c0-cb9871af938d/exporter/0.log"
Sep 09 12:56:49.362887 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:49.362865    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vl2k6_3c86c595-fb8e-45e3-b6c0-cb9871af938d/extractor/0.log"
Sep 09 12:56:49.671793 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:49.671762    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-prpjx/perf-node-gather-daemonset-9ncvj"
Sep 09 12:56:51.965406 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:51.965352    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5d49f968-hfkz2_d2536917-17ea-419b-84ee-a01daefd69e7/manager/0.log"
Sep 09 12:56:51.988119 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:51.988092    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5d49f968-hfkz2_d2536917-17ea-419b-84ee-a01daefd69e7/copy-manifests/0.log"
Sep 09 12:56:53.566016 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:53.565981    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5465bf697c-9m8xc_0d9a27ad-aebe-4c40-a579-a140aea302ac/manager/0.log"
Sep 09 12:56:53.649412 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:53.649375    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5948bbb55b-6bs6z_c0cfa3ac-1334-4119-9c97-da15303f86ba/kube-auth-proxy/0.log"
Sep 09 12:56:53.789761 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:53.789733    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8c9778b-t8sr5_11252896-9746-4dbf-9ca6-01c7d854fab9/router/0.log"
Sep 09 12:56:55.673011 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:55.672983    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5948bbb55b-6bs6z_c0cfa3ac-1334-4119-9c97-da15303f86ba/kube-auth-proxy/0.log"
Sep 09 12:56:55.818551 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:55.818520    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8c9778b-t8sr5_11252896-9746-4dbf-9ca6-01c7d854fab9/router/0.log"
Sep 09 12:56:59.042734 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:59.042703    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-pgwwm_c00ec462-3123-4de8-b39f-297d8fa14545/kube-storage-version-migrator-operator/1.log"
Sep 09 12:56:59.043628 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:56:59.043608    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-pgwwm_c00ec462-3123-4de8-b39f-297d8fa14545/kube-storage-version-migrator-operator/0.log"
Sep 09 12:57:00.204952 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:57:00.204923    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nvm9j_19401fc8-ae1c-4249-9a75-d7aab8a904de/kube-multus-additional-cni-plugins/0.log"
Sep 09 12:57:00.235099 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:57:00.235071    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nvm9j_19401fc8-ae1c-4249-9a75-d7aab8a904de/egress-router-binary-copy/0.log"
Sep 09 12:57:00.264547 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:57:00.264520    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nvm9j_19401fc8-ae1c-4249-9a75-d7aab8a904de/cni-plugins/0.log"
Sep 09 12:57:00.289987 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:57:00.289959    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nvm9j_19401fc8-ae1c-4249-9a75-d7aab8a904de/bond-cni-plugin/0.log"
Sep 09 12:57:00.316011 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:57:00.315982    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nvm9j_19401fc8-ae1c-4249-9a75-d7aab8a904de/routeoverride-cni/0.log"
Sep 09 12:57:00.338233 ip-10-0-133-152 kubenswrapper[2578]: I0909 12:57:00.338195    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nvm9j_19401fc8-ae1c-4249-9a75-d7aab8a904de/whereabouts-cni-bincopy/0.log"